Hi, B.. Your son may have hearing issues, too. Not necessarily a hearing problem, but the noise of the other children and the school in general may be a problem for him. He might have a problem with the bright lights (find out if the school room is giving him a headache), or he may be allergic to chalk dust or other stuff found in the classroom.
Something about the class room is provoking him to act out. He may simply be too immature for school just yet. This new standard of forcing kids to be in school at age 4 is not based on any kind of sound, developmental science! It's an effort to blame the bad literacy rate on the child. Unfortunately, starting a child out in school too soon tends to give the child an intense hatred of school and a life-long avoidance of learning. So VPK can actually contribute to illiteracy instead of curing it.
Your child can learn, obviously, if he works well with you at home. I'm not an expert, but it doesn't sound like he has a learning problem. He may have a problem socializing (the speech problem can account for some of that). Kids who have hearing issues, whether a hearing loss or some other trouble with being bothered by noises, have a hard time learning to speak. He may hear well enough, but the sounds may be distorted, especially if there are too many of them coming at him at once or they are too loud.
He may simply be around so many children that he feels lost or threatened. If he hasn't been in too many situations with large numbers of people in his life so far, he may not know how to be comfortable in a crowd. He may just need a smaller classroom size.
He may need another year before he's OK to be in school. In the meantime, since the two of you work well together, you can give him the same learning experience at home, and it will give him a positive foundation instead of creating a negative emotional foundation for school. Kids his age do not handle frustration well, and if he keeps feeling totally frustrated by this situation, he's not going to handle it well.
Anyway, I hope it all works out well for him, and you find the right setting for him.
Peace,
Syl