My advice to you is to stay strong and continue to listen to YOU. You know the right choice for yourself and your family, based on all that you have lived through and learned along the way. The hard part is listening to yourself, which you have already proven you are willing to do. Continue to put you & your children first. Be proud of all you have done and the strength you have found in yourself.
Your son will react to whatever he sees and FEELS, so try to keep him out of the middle of any arguements or problems. Also, sit down and talk to him. I know you said he was 4, but you would be surprised what he understands or is picking up on. Ask him if he knows why his father is not there anymore and why you moved, etc, etc to see what he has created in his mind for an explanation. At his age he just wants to know his father loves him, why he's not there and when is he going to see him. They do not care about all the other things that we as adults get to worry about!
If you want to talk, you can send me an e-mail. Stay strong!
A.