I have to agree that sometimes we have to say nothing and get away from the situation - you don't know what a confrontation would bring about and you do have your own safety, and that of your children to consider.
I've been one to shoot disapproving looks towards ones speaking inappropriately among children - and it usually works. I've also removed myself from the situation (let those people pass, or find a different route). I've said "there are kids around!" and other times that I've loudly complained to whoever I was with about the 'bad seeds' around us. Really, it depends on the situation, where I am and all. I would never do it in a way to put my kids (or me!) at risk.
Sometimes, I'll even apologize before I ask them to not swear. Am I sorry? No....but it makes the request sweeter and shows them that I realize they can choose to say pretty much whatever they want and I'm asking a little favor. I've never had anyone refuse and most apologize back and do oblige.
Obviously the door was opened when they mentioned the kids...I probably would've turned around and smiled and said Thank You!! Or said "I'd appreciate that...kids repeat everything!". In any event, your hubby is right that you don't know how people will react - so if you do say anything, be as nice (and non-confrontational!) as possible.
When your kids are old enough and this happens (hopefully it doesn't, but...), you should let them know that you hold them to a higher moral standard that it's not okay to ever say certain words. Use experiences such as these to educate your children on right & wrong and what your expectations are for them.
Best wishes....