I have an uncle that lives several states away. He makes very odd decisions about when he comes down to visit. Sometimes it's convenient for us to head to my grandmother's or my other uncle's house (15 minutes away) to see him and sometimes it's not. We have busy schedules and our jobs don't allow us to simply take time off whenever we feel like it on short notice.
I also couldn't just pull the girls from school for something like that when my middle daughter would get completely thrown off her game by the rude change in routine. She would have meltdowns from the stress of her routine being farked around with. Thank you, Autism. If it happens to fall at a time when it's convenient, we go.
The same thing happens when it's relatives on my husband's side of the family. And seriously, any other adult that throws a hissy fit will have to deal with it on their own. They choose their behavior and their reaction. I'm not responsible for that. You're not responsible for how your MIL responds either. You DON'T have to deal with MIL drama after telling her it won't work for you guys. "I'm sorry you're disappointed, but maybe James came swing by over here before he heads back. He's here for several days, after all, and we're still working while the kids are still in school. It's so much easier for you and Dad and James to make the trip here and stay in a hotel anyway. If that doesn't work for you, then maybe we'll see James on his next trip."
Your husband needs to man up.