Being around other children, especially those who are just a little older than him will help his development immensely. Not that staying home with your child is a bad thing, I would have loved to have been able to. But, the interaction around other kids means a lot. From the time my little one was in day care I could tell when he was moving toward being the oldest one in a particular class which was based on developmental level. When he was one of the younger ones, he always seemed advanced for his age. As those older kids started moving to the next class and younger ones would come along in his class, I would notice his development slowed. As soon as they moved him up, we were back on the fast track of development.
Additionally, he needs to be seeing a speech pathologist if he is not already. Children who need special assistance with any physical or developmental problem are afforded that from birth. Your pedi should have already suggested it to you. You should contact your local school district to obtain information on beginning services with the local speech pathologist now. It has to be frustrating for the child. I understand his frustration with trying to make people understand what he is saying. You and your hubby are around him all the time, so you know when he pronounces a word wrong what he might be trying to say. For example, my son called a banana a mana, a nuuu-nuuu was a motorcycle based on the sound the motor makes, lel-low was yellow (he used those until he was able to say the bigger word). I remember every time my son went to visit my parents, who live away from us, I would have to give my mom the "Dwayne Dictionary" which was an explanation of how he said certain words and what he meant by them.
I know as a parent you only see the perfect side of your child. It's hard to admit there is a problem. But, once you get over those fears of my child is not perfect and start getting him some help, you will be amazed how fast he progresses.