It's a problem not just because they live together, but because you discussed this vacation in front of him. The implication is, it's for everyone involved in the conversation. That would apply if it was a woman or other platonic roommate, frankly. If the topic came up, you or someone else should have said, "We'll talk about this another time. For now, tell me, John, what you are working on" or "How about those Broncos" or something that involved him. I think you really walked into this situation by being kind of rude to start with - sorry, I now you didn't intend that but it is the only result possible.
You've all given him the impression that he is, by some extension, part of the family since you involved him in a family vacation discussion at your first meeting.
But it's not all that different from discussing a party with someone you just met - you can take a room full of people and let them all know there's a party except for one person there. We teach our kids not to talk about dates or parties in front of people who aren't invited - why wouldn't this apply to you and your cousin???
All you can do now is say you have postponed any discussion of a get-together this year, and then you invite NO ONE. I get that you don't like him much anyway, and you don't know him well enough anyway.
On the other hand, we had a family reunion for someone's 90th birthday, and we invited the fiancé of one of the granddaughters. They didn't stay in the same room for a lot of reasons (we had a single female and a single male who each needed a roommate so we split up the engaged couple). But it worked out great because we got to know the fiancé. It's possible that would happen with you.
For now, I'd set up a few more dinners to get to know this guy and defer any discussions of the summer.