First, know you cannot change him. You can change yourself which might enable him to make changes. Even if you decide to leave, the following information could help you to be more confident in yourself.
I suggest you read about codependency. A codependent relationship is one in which one's happiness depends on the other's behaviour.
I was obviously codependent when I started counseling in my thirties. My counselor recommended I read a book about codependency. At first, I was very upset declaring I was not codependent. Gradually I began to accept that I was.
Similar to you I was afraid that someone I dated would leave/ abandon me. I dated several men who were "healthy" enough to leave. I was clingy and was upset when they didn't see me and didn't let me know they wouldn't come over when they had said they would. I would feel frantic and kept trying to get them to understand how I felt. Looking back, I think at least a couple understood and chose to withdraw because I was needy and depended on them for me to be happy; a behaviour they weren't able or willing to live with.
Of course not calling when they changed plans and being unable to talk about their leaving was not healthy. They just stopped calling which frightened me more. I depended on them. They did not need my anger and unhappiness.
Unlike you my relationships usually only lasted 3 months. Not continuing our relationship was a healthy thing for them. They knew how to protect themselves. My expecting them to understand and take care of me was not healthy. That's codependency.
Another important change for me was to realize how my way of talking to people, not just those I dated, often started because I accused them of not caring etc. A book entitled Nonviolent Communication helped me understand how I reacted in anger or with tears was a large reason we had fights. This was especially true with my adult daughter.
I learned first to start comments using I words. When I said you hurt me, they had to defend themselves. When I said I feel hurt it opened the door to the possibility of a conversation. Read the book to learn other ways to get better communication.
I also learned it takes two to fight. Once I finally accepted that I could take care of myself by walking away, I could stop a painful and angry fight by not participating. Often I just literally walked away. My daughter and I have a good relationship now that I stopped trying to fix things. We agreed that when she just wanted me to listen that I would listen. We both gradually learned how to not fight.
I don't know your boyfriend. Perhaps he goes into his cave because he is hurt, angry and overwhelmed. It's likely that both of you have difficulty communicating. I suggest couples counseling might help.
It's also possible he isn't able to work through to better communication. It took me years of counseling to have enough self esteem/confidence to change how I communicated. Change is hard work.
I suggest you start counseling with it without your boyfriend to find out if your relationship is worth keeping. I suggest, because your relationship has lasted 3 years, it's worth trying to make changes before leaving the relationship. I also suggest not buying a house and not getting married until you have better communication and know that marriage will work.