I guess I am missing something. That does not sound like the kind of damage a boy who was just big and 13 would do, someone has swung on your door, someone tossed the coats out of the closet and stomped on them, and someone broke the hinges on your cooler. Even a big rowdy kid would not cause that. Maybe the aplolstry could rip when big child sat down on it, or if they were playing roughly on the swing set it could break, but that would not be my first conclusion at all.
He may be big, and he may be breaking things, but that sounds more deliberate to me. Given that it is unlilkely that a typical 13 year old would want to play with a six and eight year old, and that he has been the one to tell you that it "has" to be this way, I would suspect that he is not really a typical 13 year old, and that his developmental social level is more in line with the younger kids, and when he does things that are of his developmental/social level, things get broken, or he may break things deliberately.
The issue really is that someone may need more supervision than you are willing to provide when the three of them are together, but if you have never actually seen him destroy something accidentally or otherwise, then you really don't know what is going on.
Either way, you should be having a frank discussion with these children's parents. You should tell them how much your son enjoys thier daughter's company, but when the three of them are together, damage is happening in your home, and you wanted to find out if this was going on at their home too. That way, you are not accusing anyone, and you also leave yourself open to the idea that the combination of your son with these kids could also be part of what is going on here. Sometimes, kids are infulenced by one another to do things that we would not expect of them otherwise, and it could very well be that both of the younger ones do things that the older child instructs them to do that they do not think twice about, given his age, as you said already, he does boss them around. The issue may just be that it is not appropriate for him to play with younger kids, not his actual size. You really need to know what is going on, or just say what is happening in your discussion without drawing any conclusions, even if your gut tells you something.
I would appropach the situation with an open mind. Either that, or sit with them the entire time and see what shakes out so that you know for sure how things are gettin damaged. You may be surprised.
Either way, you run the risk that your son might not be playing with the girl anymore.
M.