The traveling isn't so bad, but plan extra time. Your schedule may be a bit off, but when he is ready for a nap, throw him in the car and take off. He may sleep the whole way, if not, then stop, feed him, change him, and if needed, sit in the back to play with him for the rest of the ride - same on the way back.
I don't think that you are overthinking it about the germs. Doctors usually recommend that you don't take babies into crowds until they are at least 3 months.
Put the baby in an ergo facing you. If he falls asleep, you can put that head holder up and it protects him even more. Put hand sanitizer in the ergo (burts bees has one that I am okay putting on my baby), for anytime someone touches him. Don't worry about the looks you get. Also keep some alcohol-based sanitizer for those that want to hold him. You may be able to restrict it to just grandma. If you keep him in the carrier, people are more likely to not ask to hold him. I also have a baby with sensitive skin, so in some situations, I will ask that someone wearing perfume not hold her, or have a supply of receiving blankets that I also use to help keep the scent (and germs) away.
Tip - the time that most will ask to hold him will be when you eat - oh, would you like me to hold him while you eat? If he is passed out in the carrier, it is easier to eat and not feel the need to pass him, as you don't want to wake him.
This may sound a bit harsh to some people, but you are trusting that everyone that holds the baby is healthy, hasn't been exposed to anything and are properly vaccinated. Yes, they are family, and yes, you should go (I opted not to for a similar event when my daughter was 5 weeks), but that doesn't mean that you have to pass her around to everyone during flu season.