He will be fine in any environment. Typical kids are not messed up by being around other kids. Are you worried about how he will handle being around peers?
It is a it is a mistake to rule out any developmental issue because he CAN do something on a diagnostic list. Specifically, an autism diagnosis is not undone based on the presance of any one skill, and it is a common mistake parents make when they are in the mist of "you idiot for thinking that- not my kid- reactions."
Pointing is actually one of the biggest indicators, and by having this issue alone, a good pediatrician will make the referral no matter what else he can do, and if they don't you should wonder. You don't want to miss a developmental issue that would respond to early intervention on the basis of making a critical error in understanding this disorder, do you? You should also know that pediatricians have very poor records overall in sending children for evaluation in a timley manner. The point is not to avoid any overreaction, the point is that if you are the 1 on 110, you have no time to waste, so what is the harm in finding out for sure?
You are making justifications that are based on inaccurate information, in terms of eye contact, the issue is not if he looks at you per se, it is what he does with the information he gets from sizing you up while you talk, again, a language issue. It is just usually expressed as "eye contact" because most kids with autism don't use that information, so why gather it in the first place? There is no "inverse" diagnostic rule and that he looks sometimes, will in no way mean he can't also have autism.
Get him screend, your reaction is very typical of parents who see nothing because there is nothing to see, but it is also typical of parents who don't want to see what is in front of them. I would not take a risk and find out later which category I was in if that risk was that I would have a newly diagnosed 7 year old autistic who missed 4 good years of early intervention. You will kick yourself if that happens to you, and you have only the most wonderful things to gain by finding out, intervention or a clean bill of health. In my experience, parents who march in to pediatricians saying, he does not have autism because he can do this, this, and this will not be effective at telling the pediatrician something important, like what the funcitonal issue really is. You have no idea, it could be many, many things that are not autism, but need early intervention just the same. If you go in determined to prove what it isn't, you will not find out what it is- you will kick yourself later, for very good reason.
This is exactly why people should not give you a "diagnosis" if they see an issue, they should just tell you what they see so that you can ask someone who can tell you for sure who really knows what to look for. It leads parents down the wong road as oftne as it helps them seek timely intervention.
Good luck,
M.