You just have to be careful and document everything. Make sure that you save texts, etc, that clearly indicate you discussed when your daughter was available to go with her dad and grandmother.
My daughter's grandparents showed up out of the blue one day. We lived 7 hours away and they just showed up with their motor home and wanted to take her to spend the night. Their son and I were in the middle of custody proceedings with the court and I told them it was fine for them to visit with us in our home. No problem. But, since my daughter was in therapy because of being abused by her father, (he sent her home with bruises all over her shoulders and neck), AND since they had given no advance notice they would be coming, I felt reticent to let my daughter go with them.
They demanded to know the name of my daughter's therapist and left. They came back to my house and told me they met with the therapist and she said it would be fine for my daughter to go. Something didn't seem right to me so I called the therapist.
She said they went to her office, barged in without an appointment, and caused a huge scene when she would NOT tell them they could take my daughter. So....they came back to my house and lied to me.
And, in court, they flipped it around that I refused to let them see my daughter.
It ultimately didn't get them anywhere and it helped that their behavior was documented by my daughter's therapist.
The main thing is to follow everything to the letter of the court schedule and follow everything to the letter as far as communication with your ex and/or his mother. It's great when parents can be flexible, but having left an abusive marriage myself, that is not an instance where flexibility is called for. ESPECIALLY if you are in the middle of a custody proceeding. If you give in and don't follow the orders, that opens another can of worms.
Just document everything. No matter what your ex or his mother say, no matter how many people tell you that you should do the "nice" thing and let them have what they want, this is not a situation where going outside the perameters of the current court orders is called for. In fact, doing so, could do more harm than good.
If you are following court orders, you aren't in violation of anything. If you are properly communicating pick up times, places and making your daughter available at those times, you are not in violation of anything.
Hang in there.