K.,
I have a five year old daughter who is always very shy and reluctant to join in activities, and we've been doing things for years! This is just her personality/temperament. I've learned to be patient and go slow, but I don't let her refuse to participate.
Now, I'm not saying that's the right answer for your situation at all! What I would try is just taking him to the games to watch. Let him take it all in, and see if he begins to show a desire to play. My child needs to check it out and get very comfortable with her surroundings before she'll venture forth. We just started going to a gymnastics gym (not Little Gym anymore), and I had to visit with her twice and just watch the others before she would participate. Luckily, she spotted a leotard for sale that she wanted. I told her I'd buy it if she went through a whole class. She did it the very next visit! However, she is still shy when she first gets in the door.
My biggest concern with her is that I don't "enable" her to think everything is overwhelming and it's okay to always be a spectator. I'm trying to balance that with not pushing her into things she is either not ready for or not interested in. All of this is made more challenging when you are also caring for a baby!
Hang in there! I think the suggestion about trying some classes that call for parent/child interaction is a good one. Hope you hit on the right answer for all of you!!