I was in similar shoes. We signed my 4 year old daughter up for soccer, she just made the age requirement. She hated it. She would literally stand there on the field and not move. We let her stop going after 6 games.
Here is the article that I read in Parenting magazine that made me realize it was ok:
"12 Parenting Rules You Can Break"
Rule #9:
the rule: Don't let him quit music/ballet/karate lessons in the middle of the year.
the reality: Sticking with it can be torture for everyone involved.
Naturally, you want to instill a sense of perseverance in your kids, and encourage them to finish what they start. But we all make errors in judgment. When 11-year-old Zachary Schofield took up the French horn, mom Janet rethought her policy. "He loved the sound of the instrument, but it was just too difficult," says the Beaver Falls, Pennsylvania, mom of three. "After a couple of months, we realized it was a mistake." So Schofield opted for a compromise: a switch to the clarinet, a much easier instrument to learn. "We did it with the understanding that he wasn't going to keep changing instruments every few months or so, and so far he's doing great. We're all much happier."
When faced with the quitting dilemma, ask yourself: Is this a pattern? Has he asked to quit things in the past? If so, am I choosing activities that are too advanced? Before you make a commitment, do some homework to figure out if you're both prepared to make the necessary sacrifices (like getting to the figure-skating arena at 6 A.M. or devoting several hours a week to teaching your preschooler the piano via the Suzuki method).
We are going to try a different activity for my daughter. Good luck and don't let the instructor bully you around, he is your son - you get to decide.