Good morning, S.! Sounds like you have a lot of trouble there. You don't say how old the boys or their mothers are, so I'm guessing 8 or 9 years old, which would make their mothers fairly young yet. Hmmm... My first suggestion would be to stay outside with your son while he's playing and attempt to make friends with the boys. That would ease the tension you feel toward them, and you wouldn't feel so anxious when you see them. It could be as simple as offering a 'hello' when you see them or having some lemonade or cookies on hand to offer them while they're riding their bikes. Since you're the adult here, it's up to you to set a good example for your son. Making an attempt to relieve the stressful environment by reaching out positively is showing your son that it's okay to let bygones be bygones and start anew. The other boys would benefit by this too, by having an adult outside to oversee playtime (calling their mothers or 911 in case of emergency). Since you believe the other mothers have no time to tend to their boys' unruly behavior, take a stand for your own son's safety.
If the boys do not have any interest in maintaining a respectful relationship, you may have to resort to more serious measures. Have you tried calling the police to report the vandalism in your yard? I know it's not the most pleasant of experiences, but drastic times call for drastic measures, as the old adage goes. Remember, the police are there TO PROTECT AND TO SERVE. When they arrive and you have expressed your unhappiness with your ruined garden and tree, you can then address the lack of supervision and safety issues that you feel is a problem with this family. I've had similar problems in the past, and these kinds of situations are not easily and speedily resolved. It requires a mature, nonjudgmental adult to monitor the neighborhood and assess behavior and act accordingly. Nobody wants to move out of their houses, I'm sure. And kids grow so fast, why should you move just because of these issues? The problems will have been forgotten in a year or two. It's just not worth the hassle or expense.
Set a good example and extend a hand of good neighborliness. You don't have to get together for afternoon tea on a weekly basis, or even go over for an evening of cards. Your main priority here is for you and your family to be able to safely enjoy your yard and the neighborhood with peace of mind. As a sidenote, you may consider getting a video camera (or a cell phone with the video feature) to record evidence of bad behavior, expecially if the boys start engaging in criminal activity, i.e., striking you or your son, breaking windows on your house, or damaging your car! That is compelling evidence and will probably open their mothers' eyes to what their kids are really up to. Good luck with the situation. Keep us updated, please! Have a nice week. I truly hope things get better.