Your husband sounds a lot like my husband. Ooops.....make that my EX husband. I gotta tell you, I had the same exact feelings as you, and for me, the decision to GET THE HELL OUT, was agonizing. But once I made that decision, the rest was a piece of cake. My life got so much better!!! I am now basically a single mom to a 3 1/2 year old, but there is a beautiful peace to my life (or as much peace as there possibly be while living w/a pre-schooler!). I joke w/my friends that being divorced from my husband is exactly like being married to him, only cheaper. I still do all the work, but it is only for myself and my son. My husband and I did NOTHING together, since I worked full-time (he didn't work AT ALL). And when I got home from work, I had to take care of the baby (after I picked him up from daycare...don't even go there on the whole daycare issue...), make dinner, clean up, bath the baby, do the laundry, etc. I am so glad I got rid of my husband, I cannot even TELL you how happy I am.
Good luck w/your decision. Oh....one final quote..."The minute you settle for less than you deserve, you get even less than you settled for." - Simone De Beauvoir