Have you tried couples counseling yet? If he won't go, maybe you can go to counseling on your own. A good counselor should be able to guide each of you how to talk to each other about "hot button" issues without these issues automatically turning into a fight. He/She should also be able to help each of you to be able to see each other's perspective so that each of you can see what is truly going on and perhaps be more supportive of each other.
Even the best of relationships take work. Good communication and perspective taking is a skill that most people have to concertedly make an effort to try to learn; we aren't born with it. I would say to try to give counseling a shot before throwing in the towel. If, after counseling, the two of you feel that you are not meant to be with each other, then at least you have a better shot of being able to be co-parents of your daughter.
I hope that you are able to find the answers that you need.