I'd be careful about any assumptions that a need to develop social skills = Aspergers. That disorder is much more complex. If you don't see more symptoms of that disorder, then you don't need to worry about it. Also, do not abruptly discontinue his medication. Instead, if at some point you decide a change or end to medication is needed, he would need to transition under the supervision of his prescribing physician. Just stopping could be dangerous. When Adderall is too strong, it can make a child more quiet/serious. To decide if that could be happening, observe whether he is qualitatively different that way while medicated. If he is also shy after it wears off, then the medication is not the issue.
Since your son has not been "regularly invited," then if there were times he was invited, but not invited again, do your best to find out what happened. We once had a boy over who had untreated ADHD. He broke a window within 10 minutes! Another child broke a coffee table in a single, impulsive move. Sometimes the reasons are much more subtle, so you may need to gently inquire of other Moms, if possible. Just being shy does make a person harder to get to know, so perhaps that alone is the reason.
Sometimes Adderall XR wears off by the end of the school day. If symptoms arose again that bothered others, he may need a mild 'booster' after school. This has helped my son tremendously, so that he can complete homework, pay attention during extracurricular music instrument lessons, and participate successfully in after-school activities.
1- help your son realize that shyness is a choice, and not a helpful one. Many people don't understand this, but once I did, I was able to break out of my own shell and help my children, too. He is preventing people from getting to know the wonderful him, and they don't have the chance to discover what they like about him. No doubt, some people will like him, some not. That's the way it is. But if he doesn't get in the game, he'll never find the people who do!
2- because your son is already 12, I would hurry up and look for a social skills group. A group offers the chance to observe and practice skills. Your son will likely find this uncomfortable, but that is the point- he needs to expand his comfort zone. Try to have him buy into the vision of himself as being a whiz at the social stuff, and be sure he knows it can be learned & practiced, like anything else. It might help reduce the fight you might get about showing up for group.
3- don't give up! Social skills are the strongest predictor of success in life. By helping him now, he will be able to interview for a job competently, give presentations as often as needed, etc. If teachers are mentioning it, then this is huge and needs attention. It's a hard thing to tell a parent, plus busy teachers miss more subtle social difficulties. Emphasize to your son the importance of both being able to participate and be social, and be comfortable just watching. He's got one of those down, now time for the other!
4- keep communication going between yourself and your son. One of the things that caused my son to back off socially around that age was the poor choices peers were making. He wanted to keep his distance from vandals, drug experimentation, etc. It is a hard age to avoid that stuff!
5- help your son understand that there are all kinds of friends, and that he needs a bunch! One friend cannot be there all the time, and the unfair burden will drive people away because of the neediness. Instead, point out how your own friends fit in your life, i.e. a gardening friend great to work with, a workout buddy, a church friend who volunteers with you, etc. Articulate some things that you find attractive about different people, i.e., my friend, ___, is the BEST when it comes to going out to movies, she sure can pick them! But when I am feeling low, it is ___ who seems to brighten my spirits. This will help your son learn not to expect a single person to be the "perfect" friend- something he will never find. It will also help him be a better spouse someday, as he will learn not to rely on one person for all his needs.
6- get some support for yourself! You mentioned that this is taking a toll on you, so please take good care of yourself. Exercise, get your nutritional & rest needs met, and try to 'let go' of the struggle. That idea is very hard- it was for me, but I had to do it. The thing is, the emotional struggle didn't help my kids when they had trouble. It is hard to articulate this, but if I "struggled" or "stressed" in response to things, they actually felt less free to talk to me & seek my help- to protect me or not be a burden, etc. What I had to learn was how to listen and problem solve with them, without "riding the roller coaster." Instead, it's more like... I can watch the roller coaster without getting on it, and from this perspective I am in a better position to see where that coaster is heading, and if diversionary tactics are needed. heh... I hope that is somewhat clear. I also tried harder to figure out when they just needed to share vs. needed help. Sometimes I was too quick to jump into problem solving mode. Now I ask, "what do you think you might try?" If they need ideas, they will tell me. It has helped. I also focus on my blessings, my ADHD son being right up there at the top! He is a wonderful kid.
An organization like CHADD might also be able to help you find a local group with other Moms who have children with ADHD. I also enjoy relaxation/meditation stuff. It helps me be able to just watch when my son gets hyped, without going there myself! I started with a 'stress reduction' class, now I just use the techniques on my own, and occasionally pick up a new CD.
I hope this helps. Good luck with your son!