I think Zoro might realize there's nothing he can actually do against Pica and is hoping to run into some other worthy opponent he can fight instead.
Sadly, none of the badguys seem to be swordsmen. Unless maybe Diamante has one folded up in his pocket.