Reading your post, something came to mind which I am going to write here now; I don't know you, so if I'm off base, I apologize.
I think that it's not that you are 'too nice' to cut him off and that maybe there's some fear there -- that if you cut him off, he won't be responsive or want to fix things, and that will be confirmation of that 'less than' feeling you are carrying around.
Like others, I understand this feeling and have been tempted. It's hard to sit with: he's too passive to make an effort to make the relationship happen, too passive to acknowledge past hurts and IF you cut Dad off, he may be too passive to want to fight for the relationship. He also sounds a bit disconnected, like he realizes he was not an attentive father to you and so he talks about the grandkids liking him to maybe prove to you (or himself) that he's "okay" or a good guy.
I won't go into my own situation, but what you mentioned~ Expecting NOTHING~ has really helped. I'm 44 and had to learn to accept him for the flawed person he is. I enjoy what he has to offer, when he has to offer it (which isn't often).
You could consider going back to therapy, trying to figure out what it is that you want from him; how to get it in other ways since he is not fulfilling any of those important needs you had when you were young. At some point, our parents stop being "parents" (usually) and the blessing in this is that, if we want, we have the option to 'reparent' our own selves and find what we need... what was missing from our lives that our parents could not give us.
Accepting him for who he is will help you let go of the anger. His 'stuff' isn't necessarily directed at you. He probably looks at you, feels those parts of his life he failed in, and so being around you may be very hard for him. You may be a reflection of the way he didn't rise up to be the person he had hoped he would be.
My own dad practices a lot of avoidance. I've learned over that time that seeing me is hard for him because he really failed me when I was young. (He abandoned me with a mentally ill, abusive parent.) So, that may be a facet of the relationship that is getting overlooked-- for him, it may be far less about *you* and far more about his sense of self that's causing him problems.
I hope you can find a good place for yourself in all of this. You, too, can go the passive route: zero expectations, when you get together great, if not, like Momma W's excellent answer: your community and friends will make good 'family' for your own kids. We have gone this route with my folks with a great deal of success. I have a lot more peace since I have accepted that while I deserved better, I got what I got and now it's up to me to do better for my own son. It's all I *can* do, you know. Keep on going and just keeping my heart and vision fixed on keeping my own family happy. That's really all any of us can do in this sort of situation. Good luck to you.