I'll bet YOU are more worried about him being without you than he is. Being comfortable in someone else's care is an important thing for kids to learn; it will make both your lives easier.
My son attended a co-op preschool, so parents each took a turn as teacher's helper. There was a girl in the class whose mommy dragged out goodbyes and sometimes never left. Allie would cry, mommy would come back. Allie was settling down, mommy would poke her head in to check on her, Allie would start crying. She was the most insecure little thing! All the other kids were fine with their mommies leaving (after the first week, at least) and had a great preschool experience. Poor Allie had a tough time.
When you drop your son off tomorrow, say, "Bye, sweetie, I'm going to run a couple errands while you're in your swim class. Do what teacher says, I'll see you when you are finished." Give him a hug, and leave. Don't hang around, just be sure to be back a couple minutes before class ends. Tell him how proud you are of him for being such a big boy! I bet you'll be amazed to see how well he is doing!