Yes, you need to have a discussion with him. But focus on explaining that to you, it IS "a big deal to go get them and take them back," because you are already hosting, cooking and taking care of the kids. Tell him you feel overwhelmed and stressed and need his help. If your husband is anything like mine, they love to hear that you need their help and the opportunity to step up and save the day for you. Instead of being angry, frame it more like you're reaching out to him with your problems.
Don't bother trying to convince him his mom is manipulative. It's a waste of time. You can talk about how it hurts you that he isn't more sympathetic to your concerns (as opposed to his mom's), but don't make the whole conversation about his mom. You won't get very far. Trust me, my husband doesn't even get along with his mom and is the first one to say NO to her many requests. But like your husband, he swears she "doesn't realize" she manipulates, and that if she continually pushes us about some issue or another, it's just because she "*forgot* that she's already brought it up" (hahahaha).
Above all, enjoy your holiday. It sounds like things are going your way. You're hosting, and since BIL and SIL aren't coming, you don't have to drive at all. And like the others have mentioned - I would skip the Saturday Tday at MIL's. One Thanksgiving is enough! If hubby wants to go, let him - but I would stay home and relax!
Good luck!