I understand that vaccines are scary, and I think that you have every right to question whether your little one needs all of the vaccines that are out there. That said, I think that you are projecting your anxieties about being pregnant in general onto your experience with that one vaccine, and it's translating into anxieties about parenting. It is infinitely more dangerous to your baby for you to come down with measles, mumps or rubella when you are pregnant than for you to get the shot. You did the right thing. And your argument "I survived the rest of the childhood illnesses" is faulty because if you didn't, you wouldn't be writing this post. That's like saying "when I was a kid, we only were in carseats until we were 1, and I survived, so I'm not putting my kid in a carseat." I mean, you can make that argument, but why would you? We now know more and have more technology available to us, so why not do what has been medically proven, not speculated, to protect our kids?
Clearly I am a huge believer in vaccines. Still, I "opted out" of the vaccines for diseases that seemed incredibly unlikely. Like Hep A. I'm not travelling to India with the kids anytime soon, so they didn't need it. I also think that a slightly delayed vaccination schedule can be reasonable as well. I didn't get the Hep B in the hospital. I figured my kids were just born, lets give their bodies a break. They did get it at about 2 weeks old.
Finally, you said you don't take the baby to neighbors or a daycare center, but does he see anyone at all? Go to the grocery store? Babies put EVERYTHING in their mouths and on their hands, even under the watchful eye of the most vigilant parent. I got my kids vaccinated because I knew I simply couldn't live with the guilt of knowing that, if they got horribly sick, there was something I could have done to prevent it and I didn't.
My best advice is for you to have a good long talk with your pedi. about your concerns. If he or she won't take the time, find another doctor. You are responsible for making the right choices for your child, but you need to be armed with the right knowledge, not only the internet-fueled anti-vaccine hysteria. Ask the doctor to point you in the right direction for some peer-reviewed literature that supports a delayed vaccine schedule. Dr. Sears is a good place to start. Make informed choices, not choices out of fear.
Good luck with your decisions.