I know what you mean.
After all, he even killed that ant that was supposed to be better than average, with ease and even with letting it make their moves on him... to the point he even let himself inject some poison.
How was he so over confident at the time is a mystery... or not.

I guess the very difference comes from Nen powers. Killua never rationnalized that power before he started training it with Gon/Wing.
But deep in his body and subconscious, he had been trained to fear it deeply by his brother. His brother did a good job instilling utter fear, without waking Killua's nen powers. I wonder what were his motives. Was it so that he isn't exposed to threats he would not overcome due to inexperience and young age, so the family needed to seal his powers and wait for Killua to overcome that barrier and show his monstrous powers?
Or was it just jealousy, brother sick love?
Maybe a mix of all that.

Then of course, there's the pitou meeting. I guess he felt an aura that triggered the instilled fear. And now he meets people with really strong nens and that fear is still there. He now needs to overcome that hurdle, the fight is more against himself, rather than against Shoot. And of course, the story is made so that Shoot is in a similar situation, having to first fight his fears so that he goes another level.