This was asked in another forum, here's my explanation:

From what I can tell from the raw manga, it was a mistranslation.

From the raw manga, where Jiraiya supposedly mentioned a "4th master":
JMjkQ...
gdK.

My translation:
If you get it, you can be like the Yondaime (Fourth)... grasp it yourself.

Alternate translation by AK of Troy (Toriyama's World):
You're one person that can get close to the Fourth, do it yourself

EDIT: It seems that I can't post Japanese characters on this forum.