He is only 5, and a lot of things look scary at that age. Get on your knees and imagine the perspective difference. If it starts to become a problem in every day life, it is time to address it.
If he isn't having panic attacks and getting unrealistically upset, just saying no, I would see if he would be interested in trying it with a friend when he is a bit bigger or see if he will do it with you or even tell him this is a guy thing... grab a guy, your so, dad, grandpa, uncle. Boys respond differently to men.
I had a major phobia until this past year, and it came complete with panic attacks, which turned into either a fainting spell or a full blown asthma attack. I am finally ok, still nervous, but not in major distress over it. If he is doing that, you need to address it before he gets to be my age :)