Does your son's father live with you? If so,it's up to him to stop the drama. The two of you decide what will work for your family and then he sets the boundaries with his family.
It would help, tho I know it will be very hard to do, to find a way to feel compassion for the mother of the 3 year old. She and her daughter were abandoned by your boyfriend who made a baby with you. He did her dirt to be blunt. He owes her much respect and child support. Is he providing for his daughter financially and emotionally. If visits at your house are not working he needs to find a way to spend time with her somewhere else.
I suggest that there are 2-3 sides to this story other than yours that you are not considering. Please take a step back and listen to the others in a sympathetic way. You be the calm, stable mature person. Set your boundaries without being critical or blaming others. Encourage your boyfriend to be kind and sympathetic to the ex who is the mother of his child.
I don't understand why the grandparents can't have both children at their house at the same time. You say the three year old calls you and her father names. She's 3. you can stop that. Treat them as you'd treat "bad words." A time-out perhaps. And then give her lots of love and attention so that she realizes you're not what her mother says.
I suggest that, tho it's very wrong for her mother to be using those names in front of her daughter, she may have good reasons to feel that way. The best way to stop the rancor is to be nice, be sympathetic, work on understanding what is happening from her viewpoint.
Like it or not this other woman and her child are a part of your family because they are a part of your boyfriend's family. Cutting them out of your life will not work. You have to find a way to accept them as they are while setting boundaries that work for you as well as the rest of the family members. Co-operation and compromise built on compassion will help a whole lot.