A lot depends on whether you have legal custody, or whether there's just an informal arrangement that you are raising the child because her mother and father were unable to. You must have something, if you are in charge of her medical visits and so on. It also appears that the bio father, who never signed over his rights, has grandparents who cannot accept the situation and feel that, unless they can keep you hopping, they may lose a connection to their great grandchild.
She's 4, so she doesn't have much of a social life other than what you arrange. But the great grandparents will find that, over time, she resents them for interfering with her schedule. I'm not clear - are you saying that this 4 year old has a ball schedule? That may be hard for others to understand, so they may feel that you are making up schedules or putting up barriers. They may also feel that they are getting older and don't have as much time to watch her grow up as they would like.
I think a compromise might work - set up a defined schedule that works with your family's schedule, and so that the great grandparents feel secure that you are interested in having them as a part of her life. But make it work with your family's schedule too - you don't want her missing family time, vacations and holidays. And how do you think she does for long periods of time? Does she do okay with them for a week, or would shorter vacations make more sense? How does she seem when she goes off with them, and how does she seem when she comes back? Are you confident in their ability to care for her competently? Do you have any concerns that they would not return her to you at the end of the visit?
Look ahead at the summer and some 3 day weekends, suggest a schedule to them, and write it down. Tell them it will be far less frustrating for everyone if they know exactly what time they have and they aren't constantly texting and calling about it. If they know when they will have her, they can seek out entertainment and better bargains on things like Disney On Ice than spur-of-the-moment purchases that may not work out. But stress that this little girl needs stability, and that means no get-togethers without planning, no pulling her out of family activities designed to give her structure, and so on. You can have her make a drawing for them or send them a little craft that she made - if you do this frequently (and kids always have some creation or another that is crowding the refrigerator or the bulletin board), it may reassure them. And it's contact by mail, on your schedule.
What bothers me about their actions is the whispering in her ear. Whether or not she is your bio child, adults absolutely never whisper in a child's ear about plans or about who's not letting the child go where. Since they are capable of driving 2 hours at the drop of a hat, and since they text, they are both mobile and tech-savvy. That means you can have them Skype with her (vs. just talk on the phone) which will let her see their faces more and also allow you to overhear what's being said. You can allow more frequent but shorter conversations. It also bothers me that they are going to the child's bio mom and complaining. That is so inappropriate.
I think if she alternates visits, going to them part of the time and having them come to you in between, it keeps the visits frequent but also lets you monitor half of them because the great grandparents are under your roof. If they want to take her out for a bite to eat, set it up ahead of time so she KNOWS what to expect. If it's sudden, it looks like the families are competing, and that's not good for her.
I don't know if it's worth saying to them, "Look, this child is going to grow up and figure out that neither my sister (her mother) nor your grandson (her father) were able to care for her nor wanted to be in her life on a daily basis. She's going to have a lot to deal with at some point. What she needs is a strong foundation of family, and not to feel like she is a pawn in a chess game. If you two want to be our partners in giving her stability and the knowledge that she has a loving and permanent home, great. I will help you do that. What I won't help you do is create any feeling that she doesn't belong, or that her situation might be disrupted. That means we work together with mutual respect and consideration. You don't talk to her about me behind my back, I don't talk to her about you. You acknowledge that my husband and I are in charge of her schedule, and that doesn't get disrupted because you happen to come into town." But you have to have your legal ducks in a row before you say this.
Is anyone but you supporting this child? Perhaps the great grandparents would set up something for her college education? That would always be their legacy, their gift to her. But if they are unreasonable, it would look like you are after their money and not their involvement.
But I would say to trust your gut. If this seems like too much control, and they are not content to play with her because "it's not enough", then this is more about THEM and not about what's best for the child.