You probably missed my earlier post, but let me ask you this directly: If you were a badass fighter with strange powers, would you like to spend your days and nights (since it's so isolated) in a damp, smelly, scum filled, boring (since nothing ever happens there, except right now), dreary, and in every way unwelcoming prison guarding a bunch of hoodlums that do nothing but throw insults at you? Or would you like to seek opportunities in the vast world offering any kinds of environments, challenges, and luxuries and where strong people are always needed (and thus well paid) by many different factions you could choose from.
You must really like depressing places if you chose the prison. Fortunately Oda thinks otherwise and thus the major people among the guards are just the kind of rare weirdos that would want to spend their time there. The rest are nonconsequential grunts in the world government service who don't possess any special skills that would allow them to pick their work places better - and so I'd assume most of them are only waiting for a transfer to a nicer place.