Just as a question to think about: If this were a scenario where your child had Type 1 diabetes, and it was a birthday party, would you go? Would you take your kids?
We as parents have to balance letting our kids enjoy life, while protecting them, and unfortunately in this case, the threats are very real, but apparently aren't taken serious or understood by your family.
What you could do is talk to your Dr about it, and get him/her to write up a short letter explaining the issues (or get them to give you an article about it--seems like Dr have articles on all sorts of stuff that they hand out). Then, send a letter to the family saying something like, "sorry that our's sons allergies have caused some issues in the family, but they are very real, and very serious (life threatening). We want to continue to participate in family activities, but we don't expect people to bend over backward to avoid the allergens. What we are asking is that, if you're planning to serve food, you let us see the ingredients list if at all possible, so we can prevent dangerous (life-threatening) exposure to allergens. If we can't see an ingredient list for foods, we can't let Timmy share, and it's very hard on him to come and be able to participate in that way. Please read the enclosed, so you can see how serious his allergies are. He's allergic to X, Y, Z. Again, we don't want you to bend over backward or throw X, Y, Z out of your house. We are just asking you to help us to protect our son's health and well-being, as we would do if your Suzy or Billy came for a visit. We realize that we are first and best protector's for Timmy at this age, and we appreciate any and all help you can give us, as his loving family. We look forward to seeing you at ABC Event."
That being said, if all of the allergens will be present, and everyone will be trapped indoors and exposure's pretty much guaranteed to the allergens, I'd say no to going unless he really wanted to. And if he did, I'd have his meds right there, just in case. Otherwise, I'd probably wait for a less risky event to go to and go to that instead.