When my daughter was young and seeing her Pediatrician, I made it a point to tell them that my father is allergic to penicillin. They wrote it down, and every time she needed meds (she needed meds a lot; she was always sick), the fact of penicillin-attack came up, and she was always put on sulfer drugs.
In fact, we did try her on Amoxycillin when she was two; she developed something (I can't remember what now, probably a rash), and the doctor took her off it right away and put her on something else. It was over a year before they tried any type of penicillin again, and then she was okay. We're not quite sure why the Amoxycillin didn't work the first time, but suspect she might have had a secondary infection that mimicked a reaction.
Anyway, the point I wanted to make is that a doctor should take any type of reaction--especially to medicine--very, very seriously, especially in young children and babies. If it were me, I would find a different pediatrician. I'd want a doctor who was interested in keeping my baby safe. Doctors are NOT God. And they shouldn't pretend to be. If he never apologized now, is he going to apologize if your son dies because of an allergic reaction? I'd think stronglly and deeply about this. No doctor should be so cavalier about something that's so potentially life-threatening.