we aren't saying that he's lazy in general.Just this,

A)he isn't a power house,a requirement for Kage.

B)putting him in the position of a kage,would make him lazy.

He's just not that type of guy.