J.,
This is a hard decision I think for every mom! Unfortunately the world we live in today is a very rough one! It's scary and you hear terrible things all of the time!
I think that you should always be in sight of your son. Sit on the porch and read a book and let him play outside. If you can look out your window and see him in the backyard that's a plus! If you can supervise your son from inside and he's in a fenced yard I think that's absolutely fine! Otherwise without ANY supervision I'd definately say he's way too young!
With my son I think he'd have to be close to ten before I'd allow him to be outside unsupervised. Even if you think you live in a safe neighborhood you still never know. It's not at all about trusting your son or how responsible he is. You never know what someone might do. You also never know if your son could get hurt and not be able to aqcuire the help he needs!
It's just my opinion but ultimately the deicison is yours and your husbands! Good luck...Mom really does know best and I'm sure you'll make the best decision for your family! :)