When my daughter gives a birthday party she specifies on the invitation who else is invited. Even tho my daughter was turning six she asked parents to stay if they wanted and said that siblings close to the same age could also attend if the parent wanted them too. The party was in a park. And my granddaughter has cousins ranging in age from a couple of years older to several years younger. They were invited. Mostly the older kids went off to do their own thing on the playground. Parents supervised their own child/children.
My granddaughter has gone to birthday parties since she was 6 without her mother. But my daughter or I knew and trusted the parents, the group was small and the party was in their home. ie: it did not involve transporting the children.
Someone else asked this question some time ago and I discussed how to get to know the parents and listed some concerns to address before leaving your child with anyone. You could look that one up.
I recommend that you not leave a 3 or 4 yo alone at a party unless you are friends, ie. more than casual acquaintances with the party giver or one of the mothers that will be there. At that age the child isn't able to protect themselves if things get out of hand. They can be impulsive. And because they will not know the adult(s) they may become frightened.
If the invitation doesn't say a parent(s) could stay, I'd ask and then decide whether or not I felt that I would be comfortable being there and if I wasn't to stay, whether my child would be safe given any possible circumstances.
Unless sibs are directly invited I would assume that they are not invited. It is a special day for the birthday child and if your child's sibs don't know them their presence will only add to the chaos.
I don't think my grandchildren have been invited to birthday parties when they were 3 or 4. The parties that we had for them included family and adult friends.
In all decisisions involving our children we need to consider first the safety of our children. I don't think there is any such thing of being overly protective. If we're not comfortable with the situation our child should stay at home. As the child ages they are able to handle more things on their own.
Now, we can be overprotective of teens. They need to learn self-reliance. We can't protect them from everything. Because of their age and experience they can look out for themselves in many situations.