Just be consistent and put it on everytime he goes outside. Or ry a jacket with a hood. Or try a different hat - my kids >hate< the ones that wrap around the face and velcro or tie under the chin. Maybe it's scratchy. Maybe he has no idea what it is and wants to take it off to look at it - give him opportunity to play with it in the house so he can investigate it. Maybe he wasn't as cold as you were... my kids always tend to wear fewer layers than I do when we're outside in winter because they are running around sweating up a storm.
Or maybe just try to decide if it's a batttle worth fighting. Contrary to an old wives tale, children do not get colds just because they are cold or aren't wearing a hat. So what if he takes it off. Eventually he'll learn that he's warmer with it on. Remember the 10 year rule for picking battles -- if it won't matter in 10 years (not wearing a hat in 50-60 degree weather) then don't worry about it. :-)
And it hasn't been >that< cold around here (not compared to winter anyways). My daughter refused to wear her coat to preschool today... no big deal... I insisted on taking it along. Good money says she'll wear it when they play outside when she gets cold. Kids do learn if you let them experience natural consequences (no coat/hat = being cold). Heck... my son is still wearing shorts to school (with a sweatshirt or jacket) because he says he's not cold when he's outside (too busy running around nonstop). When he starts to get cold, he'll start wearing jeans.