Yeah, and Naofumi is a hero as well, so why would they believe he's a criminal? Those four are basically the outsiders. However, they knew from the beginning the Shield Hero is the odd one out. How convenient is it to accuse him immediately of something, to throw him to wolves to get his portion of the stipend as well? It's one of those cases where nothing makes sense, like why Naofumi would still have been there, robbed clean to boot, if he had attempted a rape, yet let the victim escape? Well, it's basically the same as the Japanese police: When a crime happens, they arrest the most obvious suspect, and then pressure a confession out of the poor bastard or if that doesn't work, the court of justice will figure something out. Just like Bill said earlier, so it makes sense Japanese authors wouldn't think anything is out of place in a scene like this.
Naofumi shouted it's his armour, so unless they have gone deaf, they would know.