My daughter (5) was beating up her 3 year old brother. She has almost completely stopped.
I did four things: we had a talk about if she wanted me to hit her when she hurt him, I.e. we treat others as we wish to be treated, i stopped letting them play alone together, i stopped letting her have candy if she hurt him, and lastly, social isolation in her room for 20-30 minutes. If you can't be nice, you can't be with others. She even ate lunch alone one day.
You need to start loving him, like really loving him. By this I mean, you need to start building him up by just giving him random hugs. Compliment him when you can, and spend alone time with him every day. This will help you feel more warmth towards him, and this in turn will make him want to please you more.
You need to stop the behavior and teach him other things to do. So, instead of yelling, ask, "what happened," then discuss three other options. Ask him to list them, if he can't, offer him suggestions. But mostly, don't let them play alone. You need to prevent, teach new habits, and give him other options. This, and more alone time with mommy, is what's really needed.
So, sit down, tell him that you will no longer let him hurt his sister. Ask him how'd he would feel if others treated him that way, and then ask him for his help to help you to help him. Have him come up with other things he can do when frustrated, clap hands, stomp feet, etc. then lay out the new rules and stick to them.
I don't know why your son hurts. You need to find out what need it is serving and find replacements for it. With my daughter, she is impulsive and aggressive when frustrated. So we are working on the bigger issue of how to deal with frustration. Remember, he is only 5. Hitting is his way of meeting some need. You need to find out what this need is and then teach him more appropriate ways to meet it.
I hope this helps you. I was going nuts with my daughter some time back, but we've gotten things mostly under control by the approach I listed here.