no one can tell, he has his hat on all the time in public, and the time without it could have just been him relaxing without it on. since it's such a symbol of the hidden leaf, i'd assume he would in any sort of official thing or if there was a war on or somthing.