I don't think this is ADHD related either. If any young, young child is acting out like that, it would annoy ANY person, of any age.
My daughter, when she was that age, would sometimes have outbursts at home. It was annoying.
I agree with all the previous posters that this sounds pretty typical and normal for siblings. And your daughter so far sounds reasonable. If I were her, I might hide out in a room too. Now, there could be some resentment because dad has a new family and new baby and he lives with her and she has to commute. If that is happening, dad has to make an effort to spend time with his daughter on the weekends he has her. (Oops, I see he has alternating weeks. I hope they aren't using her to babysit him.) The GF needs to accept this. I don't know if she hates her brother. Only you'd know.
I have a 22-yr-old step son, 12-yr-old son, 8-yr-old daughter. My husband spends the whole Sat alone with my step son and I'm OK with it. They need their time alone. His son didn't grow up in our house (long story) and the time they spend together is crucial in helping the young man become the good person he is. Your husband needs to understand and appreciate the maturity differences of each child. 11 is still very young. She can handle some responsibility, but she needs to be spending time with dad first. Then maybe, she'd be more willing to tolerate little brother.
How to talk to her:
Ask her to describe her days there. Hour by hour, so you can get an idea of how it goes since you aren't there.
Ask her how she feels about the current flip flopping. (There have to be books out there on successful co-parenting. You and she and he are not the first to experience this.) Ask for her input. How does she see this working? What could be improved? Maybe read SANDCASTLES (that book on divorce) for tips on co-parenting.
Her getting in trouble because she is mean to him:
The primary problem is HIS behavior. She is reacting to it and the parents are reacting to her. Unfortunately, it will keep happening unless they get to the bottom of it. Why is he acting out? What sets him off? That is up to his parents to fix. If you can find a way to communicate that calmly to your Ex (and advocate for your daughter) that would help.
Where are the parents when he is annoying her?