I completely, completely empathize with your situation. I've been going through something similar over the past year except the boy next door is 2 years older than my son, which exacerbates the attitude problem even more. The parents are completely negligent and never seem to know exactly where he is at any time, which makes me totally nervous to ever send my son over to his yard.
In the beginning, when my son first began playing with this boy, I would never know what hour this kid would show up at our door - dinnertime, 8:30 pm, 9:00 pm. Once he wanted to come in and listen to bedtime stories. Another time I caught him peeking in our windows during a brunch with friends. To make matters worse, my son loves playing with this kid, probably because it's a big boy paying attention to him.
I put up with it, put up with it, put up with it, tried to build a rapport with the Mom, with the babysitter. I finally had to put my foot down after numerous incidences (the kid is a trouble magnet and, as I mentioned, completely unsupervised at home).
Now my rule is that the boy can't play in our house and my son is not allowed over in their house or yard. They are only allowed to play together in OUR yard. Neither like the rules very much and they're constantly trying to bend them, but it has made my life significantly easier and has completely cut out the after-hours visits.