B.C.
Your father - after all this time - is still acting like a child - and it's not your job to raise him.
At this date and his age - I seriously doubt he has the ability to grow up.
If that was all there was to it, sure, go ahead and talk to him every so often.
But he is too much of a child himself in order to have empathy for anyone else let alone his own grandson.
You're a parent now - and in most cases that makes you grow up in ways that are pretty good for you - too bad that never happened for your Dad.
Since you are raising your son - it's totally alright to cut off relations with your Dad.
(In your Dad's world, I wouldn't be surprised if your Dad feels annoyed at the competition for attention your son is providing.)
Your Dad can't/shouldn't be the center of your universe.
Just let go of any feelings of guilt - you have no reason to feel any.
Get some therapy if you need to but it's time to leave your Dad behind.