A. darling... let me start off by saying you are not alone at all. My oldest will be 4 in September. He has started getting more pushy with his little brother when he doesn't get his way. YOU ARE RIGHT! It IS frustrating. There are days I wonder "am I making my little kid miserable?" You are not. Just teaching him boundaries. That is soooo important.
What I have found that helps me most when we get into what I call "death spiral mode". Is stop... collect my thoughts and try to build into my child. Positive reinforcement makes the biggest difference for both of us. Give him a little job and praise the heck out of him when he does well. (give him a rag and let him dust. pick up toys and put them away. Help you put lunch together, water the flowers outside, etc). This is a self-esteem building exercise. He needs to know he makes you proud. It's the "knight in shining armor" syndrome that is naturally built into men. We need to nurture that. When he goes wayward. Take a bit of time to let him know what a good boy he can be. I've found it eases the violent behavior a bit.
If he starts hitting. Throw him in time out and take a second to collect yourself. Once you make up, talk about it. If he's still being a pill. He's probably overtired and needs to go off to beddy bye for a nap. It truly aggravates me when people are so quick to throw a diagnosis of ADHD or Autism on a child. For crying out loud... it is growing pains. LIFE HAPPENS!! I'm sure your baby is fine. Just needs some extra TLC.
My three year old starts hitting when he"s frustrated and can put words with his emotions fast enough. JUST like you talked about earlier. He may know you TELL him not to hit, but registering WHY may not make sense in a little kids mind. Give him... and you, a little grace. He'll get there. Now that I've said all this, I'm off to go follow my own advice. ;) Best wishes!