You both need to enter marriage counseling. It sounds to me like perhaps his mind is elsewhere, maybe work, maybe he has other issues going on, or he's the kind of person that likes to ignore problems and hope they go away, rather than confronting them, or having someone else deal with things. Some people just hate conflict and run away from it. Maybe he just doesn't know how to communicate, and assumes that if he has nothing to say, you will take that as your cue to do as you please, or that he agrees with you. Maybe it's his way of being passive-aggressive, and heck, may not realize it is coming off the wrong way to you. It could even be a case of a mental disorder, where he forgets things or is scatterbrained (ADD).
If you both love each other and want to make it work, and honestly, throwing away a 20-year-marriage over something so petty would be silly, then I would suggest counseling and some date nights alone to work on bringing some romance to your marriage. I don't see the harm in getting your own hobbies and friends as some suggested. Maybe once he realizes you're pulling away, it will make him realize something is going on and he'll want to talk, or be included on the outings, rather than sitting home alone. If not, at least you made some new friends, have a purpose in life and have fun without depending on him. '
Some people are just very independent. I have some female friends who only spend weekends with their boyfriends, and they spend weekdays with other women or people of both genders attending movies or having book club meetings, which their boyfriends don't really care for, or they are busy with work or spending time with their own friends. They have been together for decades and both are happy and in love. Absence makes the heart grow fonder, I guess.
Maybe you both need some space, and then come back together strong. I can't tell you what to do or what would work in your marriage as I don't know him and if he's the type of person who is more traditional and controlling, expecting the wife home all the time and cooking a fresh dinner, doing all chores, and would flip out if you assert yourself, making your marital problems even worse. For some people, it works, for others, it's a divorce waiting to happen. After counseling and seeing whether or not he wants to fix things or if he couldn't care less if you decided to leave, you can decide whether divorce is the only option or not.