This has nothing to do with the boss, and you know that, right?
I understand that you feel you've been burned once, and you're on edge. I'm sure it hurts that he's sweeping it under the rug and acting (at least in your view) that you're an "idiot".
I guess I'm really bothered by your suggestion that "he's a good guy BUT...." and your thought that people are human and "might do something stupid and wouldn't think." Really?? I've traveled many times with men, and my husband has traveled with women. We both have lunch with people the other one doesn't know, but knows about. But we have a strong relationship and tons of trust, and we're both mature enough to not do things without thinking. We have a relationship that's on a much higher plane. Neither one of us has ever felt suspicious or betrayed.
My husband's daughter, on the other hand, has a husband who thinks it's awful that his wife works in a company that even employs men. He thinks men and women shouldn't work together at all. They have massive problems and are both miserable - she hates his suspicion and possessiveness (he even stalks her and says horribly embarrassing things at her company's holiday parties and so forth) and he hates that she is sick of it.
Yes, he could stay home from this trip, but this will only come up again on the next occasion. And he will always resent you for holding back his business venture. Meantime, you'll feel good that he stays home but you'll feel guilty for hampering his professional goals. It's not going to work.
I'd say you both need more counseling, because whatever you covered way back when, it wasn't enough. You're jealous, and he's dismissive. That's not a strong foundation. Work on it some more.