First of I don't know why some people keep trying to find out who the baby looks like when they just born, babies are all swallow, their eyes are not the color they will be, the hair color and texture may change too.
Even (not saying it is) the baby was not his, who cares, they better then other should know a parent is the one that cares, loves and provide for the little one.
Second, their comment is also very ignorant, not all kids look like the parents, my younger doesn't look at all like me. Since your husband is adopted, how do they know if your baby doesn't look like HIS birth family.
If this is bothering you, I would don't go around and tell them once and for all, "Stop the comments, this is OUR baby and period. If you can't see it then keep t to your selfs because for me is very tacky you keep commenting about it, maybe he looks like my husband's birth family, who knows right? "
They may get upset, but soon they will let go and you guys can move on and let this behind.
I can't help to much about the family thing, my husband is very privet with every body, not only my family and the fact that my mother speak very little English doesn't help, but I do understand what you say because I did lots of effort for his family to like me and even if I didn't speak well English I still trying other ways. He on the other side, he stays quite in a corner, but in the other hand is he the first one that pays their tickets to come here, he has never tell me once that maybe they shouldn't come, he is always buying them presents and he may not say it to my mom (which I will like) but he tells me that he likes them and that my mom is nicer then his :) so I take that.
Btw, just to let you know, babies change, my girl use to have gray eyes and black hair straigh hair when she born, now she has brown eye and light brown curly hair, lol. and my older has jack nicholson eye brows and NOBODY in either side of the family has them...and I think I would remember if I had sex with jack nicholson and I didn't, so, nature is strange sometimes.