It is up to you, as the Parents, to ALSO, ask THEM what their plans are with your child. To state her bedtime. To state what she can or cannot eat.
To ASK, any questions.
YOU can also, while you are out, CALL them, and see how she is doing and where they are etc.
It is also up to you... to decide, IF and when they see your child. You said they see your child 3-4 times a week already and you feel that is WAY too much. So the solution is: don't let them see your child that much.
Say you have plans, say it is not a good time. Say something.
They don't drive the wagon. You do.
You just say, NO.
Grandparents will do whatever, unless you state specific things to them. AND simply call them... to see what they are doing and where they are, when your child is over there.
I am sure, they have CELL phones?
If not, then get them some. So you can call them anytime. From anywhere.
You... expect them to 'ask' permission for things per where they take your child. But they don't. SO.... the solution is to simply CALL them, from wherever you are, and ask them "So how's it going? What are you all doing?", or "Where are you guys? Can we speak to our daughter?"
If you want to know where your child is, simply call them and ask.