Your daughter is right at that age where this starts to happen. This has always been a hard thing for me, but at some point, I had to realize it was my CHILD's birthday, and as such, it should be HIS friend there. There were some cases where if we had invited all the siblings, some of his own friends would have to be cut to keep the party within the venue's limits. I didn't think this was fair to my son to not include his friends so that siblings of other friends could be there. The exception was if he was close friends with both siblings, of course.
As much as I would never want to hurt a child's feelings, at some point, I had to be the grown-up and do what was right for my child. So, in those cases
where there was a guest limit and/or the venue was inappropriate for smaller siblings, I talked directly to the moms with younger kids and explained the situation. I always started with "This is hard, and I feel bad about this, but the place where A's party is not appropriate for younger kids......" They were all understanding, and we still remain friends.
In one case, when youngest was turning 7, he wanted his party at a batting cage place. It was no place for younger siblings to be in front of those pitching machines, and even in the area where the kids could play baseball, it was too dangerous with swinging bats and flying balls for the young ones. I explained it, and thank goodness there were no little ones there! There were a few tears after being hit by balls as it was with the 7 year-olds!
It would be incredibly unfair for you to have to watch another person's 4 year-old (and a 4 year-old DOES need to be watched at an open gym place) and miss out on your son's party.
I think if you explain it to the mom, she'll understand. If she's never had this experience before, she may seem a little hurt at first (hopefully not), but I can assure you, this won't be the last time this happens, and every mom with younger kids experiences this eventually.
Also, at some point, kids do need to learn that they aren't invited to every party and it isn't the end of the world if they're not. It's perfectly acceptable and normal for kids to have their own friends and parties, separate from siblings. It's hard at first, but it's something we all go through.
Hope this helps and that your daughter has a great birthday!
J. F.