Next time you see the mom, ask her for a few of her time. Let her know that you appreciate her telling you about the loud noises, but you're concerned that you perhaps didn't understand what needed to be done. Ask if he has a sensory processing issues. Tell her that while you're okay with him playing with your kids, he has been rough with them and you want to be able to handle those situations as best as possible for him and for your two kids as well.
Approach her with compassion, and assume (until proven otherwise) that she's a responsible mom who would not allow her child to play with others if she thought they would be hurt by her child (ie, don't assume that she's just pushing out the door and shoving him off on you, come what may, unless you have further evidence to do so).
If needed, you can use the reasoning, as suggested by 1stTimeMommy, that it's a school week, so you'd like keep the playdates to the weekend. And on breaks, if things don't improve or you still feel uncomfortable with this, be sure to have the family be busy so that you're not around for drop by playdates every day during break.