From what you are saying here, it seems your husband has a lot of growing up to do. There is nothing wrong for either of you to have alone time with friends, but when it becomes more important then being active and apart of the family something is really wrong....
Maybe it's time to try a separation. Maybe he needs to move out, be on his own to help him realize what he is missing. Maybe he'll grow up then and decide to work on having a healthy relationship with you....or maybe not..
Go with your gut feelings. Look inside...listen. You will just know that you have given that marriage your best shot and have had enough.
You have choices.....
If you can come to terms with what he is doing and not doing and can get on with life with things as they are and I mean fully excepting the situation as is.... then do so. But this means you have to stop fighting with him, trying to change him or the situation and not be able to have him to count on. You'll need to keep house, work at your job, deal with your children, and hire the help that you need so you'll have sometime to have some social yourself. Yes, much like single parents do. And know that's what your children will grow up learning about relationships and marriage......not good.
If you can no longer except his behaviors and he is not interested in going to therapy to work and meet you half way on the problems then talk to him about moving out and how much money you are going to need from his salery to run the house with your two children. Figure out before hand what you will need and be firm about this coming first before what ever it's going to cost him to live elsewhere.
If he makes a fuss about it, stay calm, don't get into a power struggle with him. Give him a chance to work it out in his head and when he's not around call a lawyer and find out what your rights are. Don't take action on anything else at this point, just gather information on how you will survive $$ and what you can legally do to get that and to get him to move out.
When you talk to both, keep notes, keep a log on all information.
Make sure your children aren't around when your talking to your husband or a lawyer about any of this. Although they are young and don't understand what is going on, do keep in mind they are picking up and feeling the stress, attitudes, frustrations and unhappiness from both you and their Dad. No matter what direction you decide to take, work on keeping your spirits up in terms of your children...Keep them secure and loved at all times.
Keep in mind you deserve to be happy. If your husband is so not into sharing that happiness with you...then you deserve to be free to find that happiness for yourself and for your children.
I know you have a lot of mixed feelings now and are frightened but no matter what, be good to yourself, know your worth. Know you will and can survive without him. Either choice is not going to be easy. Nothing in life is easy....we pick our pain, our goals, our roads and it takes hard work, time and patience to get through it all. You aren't alone and will survive.
In terms of your question about the effects on children... when one parent leaves the family unit, being due to death, seperation, and divorce encluding long business trips. ....they will experience feelings of abandonment.. They don't understand what is going on yet they know something is changing and something is not right....they do become frightened and insecure....children even blame themselves for their parents problems.. It's all normal feelings and it all has to be delt with....not ignored.
Some couples get so angry with each other, they forget that children have feelings and they use the children as amo to hurt each other....Some couples even spend a life time bad mouthing each other to their children...it all has to be delt with properly.
But know when the relationship isn't right between Mom and Dad...it's best in the long run for them to grow up with single happy parents...or even step parents who are healthy who love them. With love, help and support, children adjust and grow up well.
Your children are very young now...and I tend to think if you devorce now...it would be easier on them in the long run....