Please send him. I write this as a parent of a child who changed schools to attend an advanced academic program at another school in the same school system. Send him. He will survive and he will thrive, faster than you or he realize. What he as a child may see as a big deal in leaving friends is -- you as the adult know this -- a tiny blip in the much larger picture of his school life.
Do not let your worries, or his, blind you to that much larger picture here. Very few schools want to admit to parents that they're not up to snuff; yet his current school actually admits that "the entire fifth grade is at a very low level." His current school says he needs challenges it cannot provide. Do you know how very rare it is to hear that? The fact they'd tell you this means it's serious. And things won't change in sixth grade; there won't be any magical change that means your son will be better challenged in sixth if he stays put with this same group in the same school.
He will be bored if he stays put. And bored kids can go many ways, none of them good. He may find academics so very easy that he breezes through everything, gets good grades, is happy as can be, and finds when he hits middle school (especially if he is in an AA middle school program) that he's blindsided by far tougher material and teachers. Or he may goof around to work off his mental energy. Or he may slack off and strangely seem to do worse in school, but not because he isn't smart -- because he doesn't care much since he's so bored.
He needs to be with that peer group in the AA program. You, yourself, write that you think "he will make friends easier there because kids will be more like him and he will fit in easier." You're right; so why second-guess your adult perception and let his temporary qualms make you consider turning down such a beneficial change? He will indeed make friends fine, and will be among classmates who are moving at his speed.
You have time to help with this transition. Call today and get an appointment for you and dad, not your son, with the counselor at the new school, to discuss specific ways you can help your son make the change. Find out what extracurriculars the new school offers -- a school with AA programs may have things like Math Counts, science clubs, robotics clubs, drama club or school play, chess club, and of course plenty of sports clubs or groups....Find out all the things that are available to him there and let him know about them. He will find friends with similar interests that way.
Has he actually visited the new school? Touring it could make a big difference. The instant my daughter saw the library at her new school, when all potential AA students were given a tour there, she was ready to change. Are there other kids making the mid-year change? Ask the counselor if he or she can set up a time for all those kids to take a tour together and meet each other, maybe.
And you should prepare to be pretty active in helping him see new friends outside school. (Fifth graders can't arrange their own social lives yet, so you may need to be proactive at first in getting to know other parents and arranging for kids to get together.) Also help him see old friends some too. Can he remain in some activity he already does? For instance, when my daughter changed schools, we kept her in the same Girl Scout troop that met at her old school; another girl made the same change and also stayed in the troop, and today (six years later) our troop is still together and still has kids from two different middle schools. It's good for the kids to compare notes and to have friends who are not "friends from school." If your son is on some team, or in a Boy Scout troop, etc. that is largely based in his old school, there may be no reason he has to leave it, unless you feel that it would keep his mind too much on missing the old school.
I have to add that we also were concerned about the change for our daughter. Our old elementary school was very warm and friendly and small and had a core of very active, involved, caring parents. It was hard to leave such a warm and fuzzy place at first glance. But that school also had academic issues and student body issues that meant it was never going to offer any enrichment further than a once-a-week pull-out class for one hour with a roving AA teacher who wasn't even based at the school. Over the years I've kept in close touch with friends who had kids there and they were frustrated with things like the friend whose son was very good in math and ended up basically being told to tutor the other kids -- rather than being given more challenging work himself, or being tracked into more advanced math that he would have enjoyed. That's just one example. It was a nice school but had and still has issues that are not going to change after a year or two, so moving was best.
Don't let your son's worries drive this decision. Send him and talk it up positively and be sure he gets a tour and sees some cool things going on. He already is proud of getting in; he's just undergoing typical kid concerns about leaving friends. Frankly he will end up with new friends each school year anyway and by middle school all the friendships shift again, so go ahead and send him -- he will be fine and he will be challenged.