Can't tell you what you should do, but here is my experience:
The issues that you have now (regardless of what they are) will take years to resolve themselves...and that's only if BOTH people are constantly working on them. They will also crop up from time to time out of the blue. :P
This is just pure and basic. It takes a LOT of time to change patterns of behavior, because they're tied to beliefs, personality, thought processes, prior experiences, etc...all of which total REACTION to whatever stimuli or event is happening.
My husband and I started working on our problems 5 years ago. If I had known how long it was going to take, the pain and misery my son and I would go through, the fights, the loss of self respect & self confidence...I can't honestly say I would make the same decision again. And whenever it's bad I beat myself up, and am torn with regret for not leaving when my son was 2, and it was apparent that this was who this man was.
It's not that he's a bad man, in fact, he's a very very GOOD man...and we have tons and tons and tons in common........BUT........where my natural response is to laugh, his is to get mean. Where I think things through, he's impulsive. Where I'm impulsive (& then like, "Darn. Well how do we fix that, now???"...he rages or denies any such problem exists, or lords about acting superior and mean). Where, where, where....The list could go on. He doesn't trust me, respect me, or think about my life, except for how I could be doing MORE for HIM. In short, even though we are soooooooo much alike, we're polar opposites. Those opposites COULD be used to our advantage, we could have been a great team, where his strengths make up for my weaknesses and vice versa, but it hasn't happened. In 5 years of trying to, it hasn't happened. The best we've managed is tolerable. And tolerable is so much better than miserable that it keeps us hanging in. Quite frankly, I'm tired, and exhausted... and he's broken my heart so often that I mostly just feel numb.
The unfortunate bit, from talking with others & with professionals in the field, is that this is par for the course. It takes years and years and years.
The question to ask yourself: Is it worth it?
None of us will be able to answer that question for you. Only you know if you'll be able to get through the bad times with your sanity and self respect intact, or if you'll be constantly wondering if you're doing all this work for his next girlfriend, or if you'll be able to put your feelings aside to take care of your son until he's asleep when you collapse in tears or if you can't hold back and do it while he's awake, or how you'll react when your son starts acting like his dad (guaranteed he WILL), or how the tension of a bad marriage will affect your dreams/goals or your son's life, or, or, or, or. Which freedoms do you hold dear? All of these questions have different answers for all of us. And there's only about a thousand more questions.
I DO regret to say that I know exactly what you mean about not leaving your son with your husband without you. I'm supposed to have one afternoon a week to go work, and INVARIABLY, I end up staying home at least once a month because my husband is raging, or incapable of paying attention. Now....there're other times where I have just had to bite the bullet and "allow them to have their relationship". These are the times that my husband is doing things that *I* would never never do...but they aren't unsafe...they're just different than what *I* would do. And different doesn't mean wrong. Even though it's soooo hard to allow them that, I have to. If it's not mean, uncaring, or unsafe, they can do together whatever and However they wish. Oy. That isn't easy though. In many ways it's harder to allow the freedom to be different than to fight against mean, uncaring, or unsafe.
Staying married DOES give you the power not to leave you son in the care of his father...but quite frankly...so does divorce. If your husband is being neglectful or violent, that goes into the parenting plan requiring supervised visitation...or you document what's happening and get his custodial time pulled. That's always assuming he WANTS custodial time, or would follow up with action what he says he wants.
All of that said (not to malign my husband TOO much), he not only tries really really hard not to be mean to us (sigh), over the past couple of years he's become really involved in his son's life, and has actually found, much to his surprise that he enjoys being around him. He even taught him how to ride a bike, plays games, puts him to bed at night about half the time...in short...in the past year he and my son have actually developed a pretty strong relationship. Which actually makes it harder when my husband is being a jerk. There are good times that keep us going, and bad times that make me bang my head against the wall, hating myself for staying and putting my son and myself through this. Is it worth it for me? Quite frankly, I don't know. I'm not saying it isn't (who knows in a year it might be the best decision ever), or in a year I may have said "Finis". Pure and simple...I just don't know.
Best of luck in hard decisions.