It's a huge problem, and not just for birthday parties. It is now the custom to enclose reply cards and stamped envelopes in wedding invitations - it's ridiculous because everyone SHOULD KNOW to reply. Even with that, the host family is still tracking people down. Rude rude rude.
I would either say "if we don't hear from you, we will assume you are not coming" - or do what someone else did - when someone shows up who did not reply, be absolutely shocked! Stammer a bit, say "Oh my, what a surprise to see you! I wish I had known you were coming!" It's tough for a kid's birthday party because you hate to penalize that family's child by not having a party hat or goody bag. That's if it's at your house. I would NOT pay for people at a venue if I hadn't heard for sure that they are coming. Just say, "Oh, I wish I had heard from you because then I would have reserved a spot for Jimmy." Believe me, those people will never do that again. However, you may get some icy stares so it depends how much you want to draw a line in the sand.
If you are emailing invitations, you can send out a quick "haven't heard from you" email to those who did not reply, and tell them if you don't hear in 24 hours, you'll assume they are not coming. But making a ton of phone calls is a hassle.
Part of the problem is, when you invite the entire class, people don' think that "just one more" will be a problem, so they just drop in. Or, they find out other people are going and decide at the last minute to come (maybe they are holding out for a better offer?).
If I can't commit, I let the people know that I have likely plans and it's just not fair to keep them hanging on. Occasionally a very good friend will say, "Look, if you free up at the last minute, come on by." But I never make it sound like I'm trolling for that sort of leeway. If I can't go, I can't go.
The only real exception is when you have an "open house" - then it's just what it says - there are no RSVPs and no assigned time to arrive as long as it's within the window you put on the invitation.