I liked the suggestion of taking him to the tree and explaining that it needs to grow, etc. A fence around the tree might be something to consider also, at least until it is stable enough to withstand some drama! You may even consider having him help you plant something else around it, like flowers or hastas or whatever might work. If he helps you plant something and you talk about caring for it, watering it, etc. he will probably be less inclined to destroy the area.
I'm sure he was just "being a boy" and didn't probably realize he was damaging the tree or upsetting you, so yes discipline is in order, but so is some education. As far as discipline goes I might have him help you collect the branches, bag them up, etc. Or you can tell him that next time he asks to go outside he won't be able to b/c he did something he wasn't supposed to do. I would make it logical and not too harsh.
Other than that, I personally think it's just fine that you let him out in the yard on his own. I let my daughter play in the back by herself and I check on her frequently, but sometimes I have stuff to do and she wants to swing!