I agree with the moms who say it depends on how you feel about the parents. I had a good friend who wanted my little one to go over for play dates, but honestly, even though she's a good friend, I knew the kids didn't play well together so I used to go for coffee and stay.
My boys were aged 4 when we dropped them. With one mom, she was terrific. I had absolutely no concerns dropping him there - she doted on them and did so many activities it was like heaven for him. My other son went to his best friend's house, and when we got there to pick him up, he wasn't even there - he was down the street at a neighbor's house. I was no ok with that, and that was the last play date there.
So it really is more about how they parent and if you're willing to let them look after your child. As well as how well your kids get along.
I have a daughter who wasn't comfortable going alone till she was 5 but then ran off happily and didn't say goodbye.
I don't think there is a perfect age. I think you will know when you are comfortable to leave, and your son runs off happily with his friend.
If there are issues (he's clingy, you're overprotective, etc.) then try small baby steps. Stay for a bit to chat (I always do just to be friendly) and keep it short. Reciprocate and see how it goes at your house. It will probably ease your concerns if you have one at your house without the parent there, to see how well the kids get along.