Hi J. -
If he is your bio son, the concern should be lessened slightly, since he'll have diversity at home. However, if he doesn't have someone who "looks like him" at home, when he does encounter *and emotionally process* racism, ir will be more of an issue. It is something that is harder to see - and harder to understand- if you haven't experienced it and it's not always as blatant as some might think.
In other words, I think kids only "don't notice" if they don't have to. For example, I appear to be white (I'm not, but that's a different matter) and I didn't know my best friend was black (and I wasn't) until 4th grade. But you can be darn sure *she* knew.
However, I'll echo your other commenters in that at age 3-4, he'll likely only notice if others at the school make a thing of it. At that age the world is small. So you probably have some time before this becomes "a thing".
Best to you,
e