It is ok to be worried about the safety of your children, but remember to not exclude them from activities because you are worried about "what ifs".. I know a mom that never let her girls go to anyone else homes. She insisted kids go to her home. Guess what? Her girls did not want to go to birthday parties or to any events without their mother. This caused a lot of crying and clinging... I am talking about even when the girls were in 4th grade! I finally had to speak with the mom and ask her to please allow the girls to go to other peoples homes, cause other parents no longer wanted their kids to go over to her house, because it seemed so dysfunctional.
Our child played with the other kids in the "hood" and they played over here. We always made sure to meet the parents. I would go a little overboard giving them information about us, so that then they would feel free to tell me about themselves. For the first visits, I would go over the house rules in front of the parent so they could add to it and also to let their child know what types of behavior I expected.
I also exchanged phone numbers so we could all get in touch with each other at any time.
Once our child was in school, we were able to meet the other parents at PTA meetings and events. Sometimes while we dropped off or picked up our kids we would visit with the other parents. We then would ask the parent first... If it would be ok to invite their child over sometime. This way it did not put the little one in an uncomfortable situation if they were not ready or not used to going to other peoples houses.
I also would hand out my phone number and let parents know if they needed me to take their children home for them, I was available to help out. They then offered to do the same. This helped us build up a network of parents helping each other.
When your child visits others and others visit your home, this is learning moment like any other. You will learn so much about your child. It allows your child to use her manners, to be helpful to others and to see how other families function.
Our child attended all 3 of our neighborhood schools, so we really became close to the other families in the neighborhood and other families in the schools. We all watched out for each others children and are still staying close even though our kids are now freshmen in college.