I had to chuckle when I read your post. A large percentage of the posts on this site have to do with children not listening. It's so usual. And it doesn't seem to get better until children are much older, say closer to adult hood and then how many moms complain that their husband doesn't listen?
I do know it's frustrating and I also know you can improve his ability to hear by doing a couple of things differently. As Hazel suggested, get his attention before asking him to do something. Have him repeat what you requested and stay involved until he gets started or even help him to do it. I know that takes time but it's essential at the beginning as you train him.
It's also important to take action without repeating yourself several times. When you repeat the request without following thru he's learned that he has that amount of time before you will follow thru. He can stall.
It's hard to understand why kids don't respond to angry words and learn to obey before those words are aimed at them. I suggest that it's because they get attention and negative attention is better than no attention at all. This is one reason that touching them, talking with them while looking in their eyes, and staying involved until they get started works. You're giving them the attention they need and want as it teaches them to do what you say.
How do you punish him? I prefer the word discipline because it's a better word to describe what you want to do which is to teach him to listen and obey. When you punish, by being upset, taking away toys, lecturing, etc. he doesn't see the connection between what he's done and what you're doing. He gets angry and frustrated too.
For discipline to teach it needs to be delivered swiftly, calmly, and related to the offense. If you've told him to put away his toys and he doesn't then taking away the toys can be effective. If you've told him to wash his hands for dinner and he continues playing with his toys then taking them away also makes sense. But if he's teasing the baby and not involved with his toys then taking them away doesn't make sense to him. They aren't related. Instead, I suggest it makes sense to separate him from the baby. He has to go to a different room.
If he throws a tantrum as you're moving him to a different room, then let him throw himself on the floor and walk away from him. Let him throw the tantrum as long as he isn't hurting himself or anyone else. Walk away so that he doesn't have your attention and so he can't hit you.
When he back talks send/take him to his room. You don't want to be in his presence when he's being rude. A natural consequence for him. Tell him why you're sending him to his room and tell him he can come out when he's ready to apologize and listen to you. When he comes out, repeat why you sent him there, ask him to repeat that by asking him why he was sent to his room. When he tells you why and says sorry give him a hug and go from there.
My daughter sends her children to their room for a variety of offenses because they aren't good company until they comply. It seems to work for them.
Talk with him first, when all is calm, about what you're going to do so that he knows the drill. Then consistently do this. He will learn. It will take time. My daughter had to walk her kids to the room at first but now, at 9 and 12, they sometimes put themselves in their rooms. They started doing this a couple of years ago.
Try to word requests in a manner so that he's less apt to feel that he needs to say no. Give him choices whenever possible. For example: "It's time to pick up your toys. Do you want to put them in the basket by yourself or would you like my help? Let's put the basket here or over here." or "Do you want to pick them up now or 5 minutes from now." or "Let's sing the pick up song." and then begin singing it,
It works best when we give kids a heads up before requiring that they do something. Warn them that in 10 minutes it's going to be time to pick up the toys or in 10 minutes we're going to put on pajamas.
Have a routine that you use every time for things that you do all of the time. That way he knows what to expect. Often, we forget that doing things is still relatively new to them. We've been getting ready for bed for 20+ years. For a 4 yo it's only been 2-3 yrs and most of that time they're not able to remember. (Our brains aren't fully developed until we're into our twenties or even thirties. Babies have no memory.)