Yes, you should just accept it for what it is- that your ex is unable to do anything other than what he is currently doing. You cannot change that. Speaking from my own personal experience, often when the father is absent or sporadic, we cannot fathom why they wouldn't want to spend as much time as possible with this awesome child! And, we are angry they don't see the same value in it as we do.
If you can manage to accept this, you will be at peace: Him not visiting your son or keeping visitation has everything to do with him and nothing to do with how great your child is. Possibly, the more you push, the less he will respond. Obviously there are some things he has to deal with and maybe he has to deal with them before he can begin being a good father. Is it fair? Nope, not at all. He may or may not ever come around, but I doubt you want them to have a relationship only because you tried to force it. At some point, they will have to establish that for themselves anyway.
I make my ex-husband aware of as much as possible and if he chooses to come- then great. If not, there is nothing I can do about it, but help my daughter navigate through it. You will be able to help your son navigate better down the line, if you don't have anger and hostility behind it. I've had to teach her how to make boundaries, but still forgive. If not, it robs her, not him. But, to teach it, I had to forgive.
With my daughter (she is much older than your son though) I had to explain my responsibility in all of it- that although her father made some bad choices, apparently I did too to end up in the predicament. I had to forgive myself for those mistakes and ask her to forgive me too...because they didn't just affect my life, but hers. I cannot be responsible for his mistakes, but I have to be for mine. You cannot be responsible for your ex not being involved in your son's life...all you can do is take whatever responsibility is yours in getting to this point, forgive yourself, move forward, and provide the best environment possible for your child, helping him navigate through as he grows older and begins to understand.
Take care!