on the other hand, we've only got to see the 'brain' fraction of his crew.
the doctor, the invisible cheetah, the lady. they were all non-fighters. most of his high-hitters were defeated by OZ (not surprising) before they could show their deal.

oh, and of course, there was the dead samurai brooke. all things considered, he was quite tough (I don't think that neither chopper nor Franky could have beaten him).