I agree with the previous responses about not covering for the dad, and that your son will figure things out for himself in due time. However, that doesn't stop the hurt at all. Your son probably feels rejected, without any understanding of this being his father's dysfunction, and not a flaw about himself being unlovable.
IMHO, it's okay to tell your son that his father has some problems with understanding about other people's feelings and doesn't do a good job showing caring attention to ANYONE, not just your son. You can point out that everybody has some flaws--maybe mention one of your own, such as a recent mistake that you made or something like that--and that his father's flaw is not doing a good job with paying attention to other people, even special ones like your son. This won't take away the hurt, but it can help your son understand it better.
You can make a point of weaving some of his dad's good traits into conversations, nice stories of things that happened in the past involving his dad, etc. People have many facets to them. It's hard to grasp that concept at age four, when everything seems black or white. However, little by little, he can learn this.
Above all, don't bad mouth his dad, which it sounds like you're already trying to avoid. As you seem to realize, that also hurts the kids. They can end up coming to the conclusion that since they came from this "bad" person", then that must make them "bad" too. Even if your ex badmouths you, your son will figure things out eventually. That happened with my kids. Their dad badmouthed me, and very quickly, my kids figured out that what he said didn't match what they had seen about me. By their teen years, they would tell me that they hated hearing their dad talk bad about me, and they were glad that I didn't do that, too.
I commend you for working so hard to support yourself and your son. As time went on, my children would tell me they hated it that I had to work so hard, while their dad has a seemingly easier life, which I really didn't want them worrying about. Once they were teens, we could discuss that having plenty of money but staying angry at other people wasn't really a better life. You probably already know that you get to experience the real blessings of raising your son. As tough as it is to do this alone, what you have with him is precious. I read a lot of love towards your son in your message, so you know what I mean.