Boy, he's really doing a number on you, isn't he?
Seems to me that he didn't tell you about the party and he's specifically excluding your son who is also graduating, so that's pretty hostile even if he's doing it by "omission". It's rotten, actually. If he "expected" the younger boys to be there, he could have told you about it. I guess I would say "Oh well, how unfortunate for you that we didn't know about it and Son #3 accepted another invitation." However, if your son feels it will go badly for him if he defies his father, he can choose to un-accept the birthday party invite in favor of a family graduation. I think that puts him in the middle and it's a bit of a bad precedent, but your ex sounds pretty bitter and controlling, so I guess your son should decide, with your help, what's better in the long run. I think he'll (or both of them will) be bored silly at a party where they aren't really wanted by SD, just demanded by dad. And it sounds like you are giving them all another opportunity to be together with all the grandparents and all the siblings (and if Dad and SD stay away, it's their loss, and they will look foolish).
Around here, kids plan grad parties on a staggered schedule so they can all go to each other's, so having your son's party on a different day from stepdaughter's is fine.
I guess I would lean toward putting my foot down as you say - not a family party, and there will be another opportunity. Then you take the high road, have everyone back at your house after you all sit together at the ceremony, and you have enough people to keep it light and make it festive. I agree that your son has to stay until a designated time and then he's free to join his friends or party hop, as long as he says a gracious goodbye to all the grandparents. If your ex abandons the whole family including his parents on that day, then he will look like the jerk. I'd go ahead and put up a banner (homemade is fine - maybe the younger kids can do it?) that says "Congrats to Son and Stepdaughter" (with their names) and if she doesn't show, she looks shallow too.
If neither of them show at your party, then don't go to the beach party. And don't force the younger kids to go. Your ex has separated the family by letting everyone know that your son doesn't matter - I don't see that you can stop him, but I don't see that you have to help him. As far as his parents go, you can just say, "Gee, we had no idea about this and already had plans. So glad we could all be together on the actual graduation."