I always suggest holding the boys back but if the girls are ready, to send them. I taught in high school and the boys who were the youngest often had issues with self-esteem, especially if they were physically smaller. The maturity of boys is already behind the girls in high school. If they are some of the younger ones, it can be more pronounced. If they are socially awkward or don't have the maturity of knowing how to act, they can really have a hard time fitting in. It is so painful to watch.
If your son will be competing in sports, he will have a greater opportunity to be successful or at least confident in his abilities if he is older. It's not so much a problem if he is big for his age. Getting your license is a huge milestone in high school. If you are one of the last to get one, it can be demoralizing as it is a very obvious label.
When I have subbed in elementary schools, the only really hard thing is dealing with immature kids for their grade. I never knew their ages, so I don't know if it was personality or age. In P.E. the smaller boys always seemed to try to act "bigger" than they were, which meant they were trying to "act" bigger, not realizing that they were showing their immaturity.
I would ask yourself where you see your son size-wise in the next few years in comparison to those around him, as well as your family genes for future size estimates. It already sounds like he might struggle a bit with his confidence.