Hi T.,
As some of the other mothers have said just because he has not hit you yet does not mean that he won't in the future.
I myself have anger issues and it became very clear when I have my daughter. Everything bugged me and on top of that I have OCD. I can now let things get messy but at the end of the day everything has to put back in a specific place, if my husband puts it away in the wrong place I would get upset. I use to yell at him but now I just do a few calming exercises and put things where I think they should be.
Now with my anger issue... I was just going to live with it but there was one day when my daughter was extremely grumpy and not doing what I wanted and I felt like I wanted to smack her (not a spanking but across the face and she was only a year). I immediately knew that I had a problem, anger will lead to hitting/shaking and so on at some point. I also suffered from depression which did not help with my anger. I got immediate help both counseling and medication and now have things under control. My daughter is now 18 months and is enjoying having a mother how can be happy with her instead of yelling at her and my husband.
Either your husband has to get his life together (helping with your son, get help for his issues, and show you that he can be a better person). He has to want to do this, you can't make him... if he does not want to you could be in danger in the future and it is not worth risking physical or mental abuse.
I would not be worried about your son growing up without a father... right now it does not seem he has one anyway so what is the difference if you leave. Also if your husband does not help take care of your son now then more then likely he won't want to have him for the weekend if you do leave your husband. Does he know that you have thoughts of leaving?
If you can not live in your house (even if you share it with someone) and feel comfortable then there is something wrong. Everyone has pet peeves but those can live with those, anger and irritability all the time are serious issues.
K.