I am dealing with this as well and personally it really bothers me. I'm very cautious about my daughter and who she plays with. We only have boys in our neighborhood other than one little girl who very seldom stays with her father on the weekends.
We only moved here last year and from the very start, the 2 boys next door are ALWAYS in our backyard. It scares me that I am responsible for them (in more ways than one- if they harm someone else's property, say by throwing a ball through a neighbor's window, it is YOUR fault not their parents!) Their parents don't seem to mind and it doesn't APPEAR to me that they are supervised when they are outdoors alone in their yard (maybe someone is watching, but from the things they do, I doubt it!!)
The little girl, who was 3 last year, would cross the street by herself (without looking) and wander over and want to come inside and eat. She was constantly asking me for a snack and candy and juice. I did feed her lunch one day after getting permission from her who said, "Yeah, keep her as long as you want. Feed her anything you want." The first time she came over and asked to play, we had NO idea who she was or which house she lived in. When we figured it out and told her father she was here, he said to keep her as long as we wanted. He also let us take her to the park on my daughter's b-day and we had only ever had brief interaction with him!! I NEVER would have let my daughter do that, and I only took her because she seems a little neglected and I felt sorry for her. It *IS* a big responsibility though.
I don't understand how, if you are living in the neighborhood and have children, you are automatically trusted with everyone's kids!! I don't feel like that and it seems to make me the odd one out. I have no idea who the parents are or what they do behind closed doors and NONE of them except one has introduced themselves to me! They just DON'T CARE... they just trust that their kids are okay. SCARY!!!! Hubby grew up around here and says his neighborhood was the same way and his parents never knew where they were- just that they were safe because they were still somewhere in the neighborhood.
The downside is, since I don't let my daughter roam around, the kids all congregate in my yard. They eat MY popsicles and drink MY bottled water when it's hot, they play in MY kiddie pool, they trample MY flowers and break MY daughter's toys. I'm basically the neighborhood sitter- for free. The one neighbor she is allowed to go play in their yard has NEVER offered her a popsicle or water, etc. and the parents have never thanked me for the bandaids I put on boo-boos, or offered to watch my daughter at their house for awhile since they have spent HOURS on end in my yard instead. They seem to take for granted that I enjoy it- or they're just jerks- who knows?
Basically, do what you are comfortable with. I felt like a real outsider last year and kind of let the boys run wild in the yard because I was afraid to say NO. This year, I just discipline them like my own because, hey, if they are in my yard, they will obey MY rules. If the parents don't like it- they can take their kids home and deal with them themselves.
Amanda