E.S.
Please know that your baby girl is at an extremely impressionable age. Her experiences in the first five years of life literally shape the person she becomes and have quite an effect on her personality and self-image. The behavior of that boy is not healthy for her. For one, it could cause your daughter to develop agressive tendencies herself (regardless of your calm interactions with her), but more importantly it could teach her that people will be agressive with her and she may ultimately seek out those characteristics (think future boyfriends/husbands). Your first responsibility is to your daughter, and her situation right now could have lasting impact on her life. I would change it immediately.
Best of luck to you, sounds like you are an excellent mother.
PS: I just saw your added note ~ if the boy was yours, immediate and consistent discipline and a zero tolerance for any kind of agressive/violent/mean behavior. My three kids learned very early on that hitting, "growling" or any other form of agressive behavior was absolutely not tolerated and resulted in them being removed from the situation. I have the occasional issue with "he pushed me..etc" but now all it takes is a reminder and they are back on track.