Meanings of words do not tend to become confused over time. They change. Language is not a static thing by any stretch of the imagination.

You are both right. But..."arrogance" has shifted to a negative connotation that implies someone is boastful and proud of their own skills. Whether it is not justified and now even if it is. Arrogance is perceived.

Kei is teasing Hikari. Only Hikari. But that does not make him not arrogant to everyone else. Kei may be stating it matter of factly, and the other members of the S.A. certainly know he's not an arrogant asshole and that he's only being matter-of-fact.

But Hikari and all they jealous people in the school think he is an arrogant asshole. Is it true? No. But Kei is very public about his teasing of Hikari. So everyone outside of the S.A. sees him being an asshole to her, and presumes that that is precisely what he is.

Like many other words used as labels, "arrogance" is applied by everyone else, not the subject of the word.