Hi M.,
ASAP - As a teacher, I have seen first hand the devastation that switching schools can cause children. I definitely would NOT wait until sixth grade. I taught sixth grade for years, and it is a very difficult age, city or suburbs. By then, your son will have friendships established in his old school and it will be more difficult to leave them behind.
Kids can be brutal, as far as bullying, etc., so you want to put your son in the best possible situation as soon as possible to avoid problems down the road.
I wish you good luck! I know it is an extremely difficult decision. But remember, children are resilient, and if he has support at home from you and dad, he will be fine!!!
Good luck,
L.