My husband and I each have friends and work colleagues of the opposite sex. Some we have met, some we have not. But we pick up each other's phone calls and the other person is always happy to talk with the spouse - that is, his female colleagues know that I answer his office phone (here in the house) and he picks up my cell if he can find it and figure out how to answer it. No secrets, and we are often helpful to the other's coworkers because we know what projects are being worked on.
You can't tell who is a cheater (or a wife abuser or a child abuser or an ax murderer) by how they act. In fact, the most successful abusers are charismatic, charming people who worm their way into others' trust. So the fact that you couldn't believe the other man was a cheat just says that you have (like many people) an unrealistic expectation of what a "cheater" acts like - by definition, the successful ones are the ones who would never attract suspicion. So maybe that's what you mean about your radar being off. You are trusting the wrong people.
The huge red flag for your situation is that you know nothing about this woman except what you have seen and what the kids have said. So the question is, why not? The other possible red flag is that he is getting to the gym 1/2 hour early - there may be a reason but I'd want to find out, if I were you, why he's not home or why he's not working out, one or the other. If he walked out with this woman, why didn't he march right over to your car and introduce you? "Hey, Mary Lou, I want you to meet my wife. Honey, this is Mary Lou that I've told you I work out with."
I don't know if your husband is cheating or not. I do know that children who say "the adults talk and talk" could mean 3 minutes or 30 minutes - kids aren't so good at that stuff. Also, I do know that things are not open and communicative between you, and I do know that you lack confidence. So I think you would benefit from counseling, either with him or without him (or some of each) with the intent of improving your give-and-take with each other.
If you don't have access to each other's phones, why not? My neighbor had a problem with her husband (similar to you - off to the gym - a friend tipped her off). The husband and the woman would park their cars next to each other, then leave the gym and go for coffee together. So my friend popped her 2 kids in the car and headed down there, jumped out of the car and said, "Hi, we thought we'd join you for breakfast." So you can do that, and you can hop up early and get down to the gym 30 minutes before it opens and see where he is and with whom. In this day and age, you can even track someone's phone via GPS. You can even hire an investigator if you want. But I think you need to start with a counselor, figure out how to broach this subject if you are so afraid to do it on your own, and get an objective ally to help you figure out what he's talking about (or not talking about).
Either way, you and he need a better way to share your lives, hopes, fears, and concerns.