First, HUGS! This too shall pass. :-)
Well, I'm probably going to get Bronx cheers for this, but I feel that my two cents from another direction might give some perspective.
My son, who turned 5 on 6/1, has been swimming in the YMCA lessons since he was 8 months and progressed to Super Pike by the time he was just barely 3. He's now a Ray. I firmly believe that kids absolutely should learn how to swim so that they can at least TRY to save themselves should they fall in a neighbor's pool, the local fountain (yes, I do know about a child that drowned in a fountain, unfortunately), or even right themselves if they go awry in the bathtub, or God forbid if they fall off a boat. They also teach basic water safety and lifesaving in the kiddie classes, so this isn't bunk. Along with this, like anything you learn, consistency is key, so we do not take time off from swimming but we do have him in another sport during the session when we can.
We always told our son that he could play in the pool but only if he did his swim class first, a reward for his doing his class. This also applied to when we would go to our friend's house on the lake; he could only swim if he had done his lessons. I equate this to "No TV until you've done your homework," which will apply when he starts school in the fall. So on days when he would refuse to do his class, he had to sit on the side of the pool and watch the other kids play; this was the advice from the YMCA pool toddler director. So I'm a hardliner on continuing on.
As for reasons why he might be upset, I have some ideas that no one else has mentioned.
I don't know about the pools where you go, but at the Lakewood Y, where he "grew up," the instructional pool is a good 10-15 degrees warmer than the lap pool. So when he went from Pike in the IP to S Pike in the lap pool, he froze and hated it, cried, refused to do the class, and I didn't make him b/c he was FREEZING. The solution was the warm vests that they actually have at most of the pools but don't leave out for people to find. I also got him a long swim suit and a swim shirt to keep him warmer during classes in that pool. Fortunately the Morgan branch is MUCH warmer.
Now, there were two other factors influencing my son's negative behavior. First, his friends were still in the lower classes and he wanted to be with them. We convinced him it was a good thing that he was doing so well, so we squelched that one pretty quickly. This could be something your son is feeling too.
The other factor was that we had always had class and then play time in the IP afterward. Well, that was fine, except he wanted to just be in the IP because it was always warm, but that was now not an option b/c there were classes in that pool after his class. So if he wanted to play, it was the lap pool or nothing. We begged for the smallest area of the IP for him to play in after his class, a little corner really, and the director was fine w/ that b/c she realized that it encouraged him to keep going b/c the play was his reward.
Finally, do what feels right for you. Yeah, I'm a hardliner, and others think you shouldn't push. In the end, you'll make the right decision for you. :-)