I totally agree with you that it's wrong for men to stray for that reason.
I think it was how you wrote it that made it sound like you weren't attracted to him and you also mentioned you didn't have a very active relationship that way, so it might be why ladies felt like you weren't interested. I didn't' say anything about that, but that's what came to my mind too when I read it (that you had less interest than him when it came to that).
Women do need sex too, but it's not the same as the majority of men. It's pretty provable that men have a higher "need" for it. I think it's the testosterone hormone. It doesn't mean women aren't interested, it's just that men are interested easier and faster typically (though, your husband does sound different than what is typical). While I'm all for equal rights, it doesn't overpower the fact that there are physical differences between men and women.
To keep my man happy and to know he's loved, I do make sure to spend that time with him when he needs/wants. That does mean that at times I'm completely uninterested sexually myself - but I AM interested ALWAYS in wanting him to feel loved, so it's easy for me to be interested that way. I like to make him feel loved (and we are reasonable with when we do this! He doesn't expect anything crazy from me, but he is well taken care of that way). I also always want him interested in me and only me, and that's a way I can do that. So...perhaps I'm being selfish. hehe. We do have a very happy relationship. He tries his hardest to show me he loves me, and I do the same for him.
I want him to know I love him and I know that's a way I can show that where he'll feel loved ("His Needs, Her Needs" is a really good book that explains all this and more...as well as "Five Languages of Love" which talks about how people interpret love different and you need to figure out how your partner feels love in order to give them that type of love).
I feel loved when my hubby sits and talks with me and listens and we have nice conversations. I like to snuggle and be close too. That makes me feel loved. If he helps me clean and take care of the kids, I feel like he cares and loves me.
If he ignores me or doesn't want to talk, I feel alone and like I'm unimportant to him. Same with most guys and being intimate. I know he feels loved when we are intimate. If he's having a bad day and feels bad about himself, it miraculously flips his day around and he's happy again and feels very loved and connected. It seems to be a fairly common guy thing.
Of course, my hubby feels loved other ways too! He needs snuggle time and me to be considerate of him as well, but he doesn't really have the same need for conversation, so he has to put forth more effort in that area in order to fill my need.
It's just when that "need" isn't met for most men, it often leads them into looking elsewhere. It doesn't mean it's right or okay, though. Back to how I feel loved...if my hubby never talked to me, and I always felt ignored and unimportant to him, I would have a higher chance of looking elsewhere to fill that void I am feeling inside. It would be wrong of me, but it's also not right for my hubby to ignore me, and he would have some responsibility for it. So, the idea is that if you fill each other's needs and love them in ways that keep them feeling full of love, they don't turn anywhere else (those books I mentioned above really are very good books that explain it much better than I am!).
But with your further explanation, it's obvious that we misread what you were meaning in your other post. But that's the explanation of why women were saying what they did. They misunderstood your situation. But even if they hadn't misunderstood, it's not right for the hubby to look elsewhere or say the things that he did! I think they were trying to suggest that a solution to the problem could be to make sure to fill his needs, and in return, he should be working on filling your needs of being loved.