E.S.
fyi - I have not read the other responses...but here are my thoughts...
Four is still quite young to be hanging out "unsupervised" -whether intentionally or not. Obviously from your post, neither time was "intentional". I would have my eyes on him all of the time, unless he is in the house or in a fenced in yard (and he is checked on frequently and unable to leave). Maybe I am a little conservative, but those little guys can move fast...and they don't have the best judgement (strangers, cars, getting lost, dangerous situations, easily distracted, little sense of time, etc).
I would talk to him about "big boy" privileges...like getting to play with his friends while Mommy talks to her friends. If he is not big enough to know that he always needs to stay where he can see you (and vice versa), he doesn't get that privilege. It will be harder on you, because then you have to make him stand right by you while you talk to your friends (or go home), but a couple of times doing that and he will probably know how to act like a big boy so that he can play nearby. He should have the privileges once he is responsible enough to handle them.
As for sneaking out of the house... again, leaving the house by yourself is a big kid privilege. He is not old enough for that yet. If he is not going to follow the rules, you have to take away a similar privilege. For example, "if you are not going to ask Mommy before opening the front door and leaving, you are not responsible enough to open the front door and go out on your own. Mommy will have to keep the door locked (lock inaccessible to him) and walk you out." Again, this is harder for you than him...but you have to stick to your guns. You may feel like you are treating him like a 2 or 3 year old, but until he follows the rules and shows responsibility, he should not get big kid privileges.
Responsibility earns privileges.