I didn't read the other responses, yet, because I wanted to be sure to share my true gut response. You two are living separate lives. I understand that you don't want to live this way. I think your expectations will continue to bring you down because the man you married simply does not live up to them.
I know couples who function in the kind of arrangement that you described. I know a man just like your husband. His wife had to learn to create her own life, one that she could be proud to live. She puts her efforts into their child, their home, her work. She supports him with all of his crazy ideas (2 bands included!!!). She puts her foot down when he asks too much of her. If it is important to him, he needs to make it happen on his own. He is grown up enough to recognize that. But he works hard for his family, and he gives his daughter plenty of hugs and snuggles. It's not the easy, dreamy fairy-tale, but it works for them. They worked hard to get there because they are truly committed. Well, SHE worked hard because they are truly committed.
When you spend less and less time doting on him or waiting for his call, he will see that you don't NEED him. This is powerful. He will either realize that you are only bringing to the relationship what he brings, and he won't like it. OR he will love it. He might be waiting for you to break through this way. Did he ever mention something like this?
Obviously this arrangement doesn't work for you. Can you accept that the marriage you entered requires you to be more self-sufficient and self-reliant than you otherwise hoped? This decision is really all about YOU, not him.
YOU should take the kids on a family vacation. Tell him about it after it's all booked, and invite him. Live on your terms.
(I imagine that this response will be in the minority. I believe strongly in the sanctity of marriage, and I wish the best for you and the promise that you made for your family. I trust that you will find the solution that works for you and keeps your family together, if you are committed to it.)