I think it is an agreement that nobody will do it or the Hyuga would attack them. The Hokage might be super strong but he would get hurt figthing the Hyuga.