K.C.
It takes a lot of work on your part, but the key is just keeping everyone around you as educated as you can, and always having alternate foods available. We also avoid dairy & nuts 100%, and mostly avoid eggs too (only recently can he start eating certain things made with eggs). So I totally get where you're coming from, especially with the dairy that is hidden EVERYWHERE!
If you know you're going to a party or family function, call and ask about the menu in advance. That way, you can be prepared to bring whatever alternative items you need. You'll also know how much, if any, food at the party will be safe for him. When my son (who just turned six) is invited to a party, I always ask the mom a few days before what they're serving. I bring him his own pizza, sandwich, hot dog bun, cake, etc - whatever is needed so he can essentially have the same food as everyone else. Most of his friends have been in our lives for a few years now and they actually know his allergies well, and have been happy to accommodate him whenever possible, without me even asking. It's really nice.
Don't ever assume that someone else has read the label on an item properly unless they also have allergies in their own family. People don't realize to look for words like whey or casein. You have to be vigilant.
And with your son, whenever you are around food served by others, make a point of reminding him "you have allergies. let's check if this is safe for you. we can't eat it unless we know for sure." He'll start to understand that he has to wait for your approval. When you arrive somewhere, make sure you tell the other moms (or the librarian, or whoever is in charge) that he has allergies and not to feed him without your permission. Always say to your son when you arrive, "remember, don't eat anything unless mommy gives it to you. otherwise, you'll get sick." It's true that the other adults won't realize it, so you have to tell them that dairy hides in things and they can't just assume something is safe.
It does get easier, and it sounds like you're lucky in that he's outgrowing it some. My son's dairy and peanut allergies are still really high, so we don't know if he'll ever outgrow them. Fingers crossed.
Whenever my son is going somewhere new or playing with a new friend, I generally tell the parents that fresh fruit is fine to give him but absolutely nothing else. Usually I'll send a snack or two if I think he'll need it.