4.5 is really very young to join anything organized. That's not to say that some kids don't enjoy it, but really, it's a lot to ask a child of that age to engage in frequent practices and games. Kids at that age aren't very good at sports anyway, so there's a whole lot of sitting around waiting for someone to hit the ball, or everyone on the field chasing the same soccer ball because they have no ability to understand different positions or strategy. If he bursts into tears, he's telling you he's absolutely not ready. Listen to him.
Concentrate on small group play - invite a friend or two over, and go out and kick a soccer ball around. Get a set of those baseball gloves with a velcro strip and a special velcro ball, so they have some chance of catching it and holding on to the ball if it even hits the glove (not contingent on the child closing the glove around the ball). Do a VERY small scale game with just a few kids running bases - not dependent on someone actually hitting a ball. Get a baseball "tee" so they can hit the ball off something stationary vs. a pitched ball. Get a small soccer net or just use those plastic cones to set up a goal line, and let them have at it. Try a croquet set - not dangerous but requires some aim and some taking of turns. Take him and a friend mini golfing. Try candlepin bowling if it's in your area - small ball, short alley, smaller pins.
Try other "new" situations - go to programs at the library, try a children's museum where there are a bunch of kids but not forced interaction. Go to the interactive exhibits or the live talks (about animals or dinosaurs or whatever). Sometimes those museums also have play spaces where kids can run around with kids they don't know, but not in an organized fashion.
There are lots of ways to build confidence - and that includes letting a child grow up and mature and come into his own. Tell your mother to back off. What concerns a child at 4.5 is not necessarily a handicap that will follow him for his whole life.
After each experience, compliment him indirectly by reinforcing how much fun it was, or directly by saying how nice it was that he reached out to share a toy with that other little boy. If you can find another shy child, suggest (but don't push) that your son invite that other little boy to join him - "maybe he'd like you to ask him to play with you." If he can become the "leader" or the more confident child, that may boost his confidence a little.
Finally, embrace him for the person he is. He needs to function in school, but really, he does not need to be a "joiner" in the sense of following the crowd into whatever sports "everyone else" is doing. Maybe he will be an artist, a poet, a novelist, a deep thinker, an inventor who is devoted to time in a lab...what's wrong with that?