J.L.
I know young kids can be fickle, but his reactions are pretty strong and alarming. I don't think kids generally shut off to grandparents unless something happend that traumatized or frightend them when spending time with them. You need to get to the bottom of this. Something happend the last time he was with them and was happy to see them. Hopefully he will open up to you, and tell you what may have happend back when, that caused his opinion of them to change so radically.
It might be something minor like they disciplined him for doing something naughty, and he resents that or doesn't understand how or why they could do that. Or, maybe they told him a terrible ghost story that freaked him out really bad. Or worse case, he saw or experienced something very unthinkable or serious...the type of thing a young child might be too afraid to share with his parents for fear of an angry or negative reaction.
The key is to be calm and tell him it's okay to be honest...you just want to know what happend so that you can fix the situation the best way possible. Be prepared to stay calm and understanding regardless of what he tells you. You don't want him to clam up or feel like he can't be honest without punishment or negativity, regardless of what his spin is on things.
Lastly, for the time being, do not force him to spend time with them, so he will calm down and open up with you about them. Hopefully it's not a big deal and will be resolved soon.