Just have the birthday party when it is good for you. You are right, your daughter is only 2, so at this point the party is more for your memories, she will not remember this one.
Then ask your brothers about setting up a time for all of you get together just to have the kids play and you parents do a cookout.. when it is more convenient. Not so much of an event as just a getting together for a meal and some play/visiting time.
Being a child of divorce, it is stressful enough just dealing with my parents issues, if I had even begun to think about all of the Aunts, Uncles, grandparents etc.. being disappointed about not "getting to see us".. I would have been a puddle.
I know my parents were always stressed about visits with the other parent, so we were always just trying to make it calmly through our time together. Visiting other relatives when we hardly had very much time with our own dad was really tough on us.
I loved all of the relatives do not get me wrong, but I am just letting you from an insiders view of how hard it already is, just with our nuclear family dynamics. Try to not add any more pressure to them.
Have fun at the party..