Hi R. ... I will give you the flip side and then you can make your decision on weather you are over reacting or not.
I am from South America, my ex-husband is from the US - he didn't speak spanish either, but obviously knew that was a big part of my culture and family as I still speak spanish to my family (yes even when he was around - and would just translate for him).
When our daughter was born, I ONLY spoke spanish to her (as I still do) and he spoke english. As much as he didn't know what I was saying to her, he was never bothered because he knew it was me loving my child.
As she got older, he too started picking up more words as he heard me teach her ... so did his family, by the way.
Now we are no longer together, she is four years old and completely bilingual .. if I say something to her in front of him, I will translate or she will ... I think he and his family understood that it has nothing to do with disrespect, nor wanting to leave anyone out, as much as it is about teaching her something that not only will put her ahead of the rest, but also help her communicate with her family who do not speak english.
I think you should be happy your son is lucky to be growing up learning two languages, which will just give him a head start in everything he does ... and yes, along the way hopefully you too will learn.
Does this help at all?