Hi T. -
Yes, I agree it sounds crazy. But I assume your child is like mine - never really gets sick? That's what makes it so scary. My son had a fever of about 103 a couple months ago, and I was freaking out, but now I think it's probably because at 4 years old, he had NEVER had a fever before. If he got sick all the time like some kids do, I'm sure it wouldn't have concerned me nearly as much.
So - the doctor was right, technically - but if he made you feel silly for being concerned, then I agree with the other poster who suggested you change doctors. Doctors should assuage your fears if you're overreacting, but they should never make you *feel* like you're overreacting. You're a mom, and that child is your top priority.
Hope he's feeling better!