Wanting to and being forced to are different things.

Have you considered how Andou felt after his world could not be understood? Tomoyo's complaint is paltry compared to that. She felt that she was "correct" and in the majority, while Andou knew he was niche, yet a close friend simply could not understand him and said as much.

In the end, the gravity dude has it right. Tomoyo felt guilty from being unable to understand Andou and lashed out.