Lots of good suggestions here.
Did your daughter, actually call your Boyfriend "Daddy?"
If so, that in itself, will cause a lot of confusion for her.
A little kid, does not know the difference between a Boyfriend or a Daddy... especially when the man is in the child's life like that and you all moved for him and that she calls his Mom "Grandma."
So many, role confusions, for such a young child.
They are not "family." This was just your Boyfriend's family.
But again, for a young child, they do not know the fiction v.s the non-fiction of it all. These are adult, constructs.
Then, does your brother even know or like, that you are moving there to him? Is he, a stable normal nice guy? If not, your daughter may again, be emotional upset.
Your daughter, is more heartbroken about it, than you are.
She is only a child.
And, if your ex-boyfriend still continues a relationship with your daughter, then when he gets a new Girlfriend, she... will... not, like that. At all. It is not even his, daughter, but just an Ex's child. Not even his.
But yet, his Mom.... will be continuing to be "Grandma" to your daughter???? What if, down the road, she tires of doing that??? And then becomes close to her son's next Girlfriend?
Too many, unfortunate, issues and problems, for your daughter to deal with.