I went through this exact same thing -- workplace day care, midday visits, and let me guess, are you a first time mom too? I learned to get over my feelings -- your son will live on to give you plenty more to feel bad about. What really helped me to get over them is that, now that he is 4 years old, he has absolutely no recollection whatsoever of any of these incidents -- I mean, he doesn't even remember going to that day care, which is a little surprising considering he was there until over 2 1/2 years old.
I think us moms are hardwired to feel guilt over everything (and it doesn't help that the director used that word, "distraught." Geez, couldn't have they said it in a different way? Way to induce guilt, sheesh!!). I wonder, do you stop by midday because you have trouble trusting the providers? Or it is more like, well you are at work, you are right here, and after all if it wasn't for you working he would be right with you, all the time. In other words, do you feel guilt over the idea that he has to be in day care at all, rather than being with you all day? I ask because this was the real source of my guilt - the guilt over being a working mom and not being able to be with my son all day.
A good pearl of wisdom that I read regarding this was, instead of feeling guilt over not being there, feel happy that you have, in this day care, an extended family, so to speak, of providers who help you when you are not present. That is a gift unto itself.