Oh, A., oh .... it's not 'just you' ... I'm so sorry.
Your son will notice. He will remember, and if you avoid talking about it, he will repress it, and trust/relationship/image-of-men problems will happen. Even if he was much younger, he would notice an emotional figure disappearing.
On the other hand, he was right that you guys shouldn't get married because of the baby ... I told my Ex I didn't want him to choose to *stay* married just because of the baby ... if he didn't want to be with *me* (which he didn't), I didn't want us to present a lie to the world (and the children) ... now, don't get me wrong, I was hoping I could change his mind about the leaving-me thing, and a couple of times I almost did, but in the end he left and I had our fourth child without him. (Our son was a gift--but was also, yes, very much an accident.)
Their dad is still very much in our lives, which is some ways easier and some ways harder (for me) than if he had gone away.
Every divorce/separation book about raising kids--every one!--says to not speak badly of the other partner if the kids can hear ... most say don't do it if the kids are anywhere within a block (whether this is because they might sneak up and you don't know or because they might sense the vibrations, well ... I think the latter ;) ). THIS IS VERY HARD. VERY VERY HARD. But try to commit to it anyhow. The more blunt books say straight out the truth that children perceive themselves to be of the mother and the father ... and they HAVE to be able to reconcile the two sources of themselves within themselves (even though, clearly, we messed up and didn't keep them reconciled out here in the non-inner-child world : ( ).
This guy. Wow. That's some negative hutzpah.
I am so sorry.
Thank you for trying to keep him in your son's life despite the difficulty.
Before he leaves the state, you have the right to sue for child support (it's way easier if you are both in the same state). If you want him to sometimes call and speak to the son you two share, you may want to avoid that ... or maybe having to pay every month will keep him aware that he has a responsibility to remember. 'Probably depends on his money status and money hangups.
Arrange everything for the defense and support of your son. ALL your decisions about interactions with his dad should be based on what is best for your son.
God bless.
--K.