And Yui did mention that she thought Hachiman was serious for a second. Her heart must have fallen to the ground at that moment.

But when I remember how much damage Hachiman takes from these two girls (even though he is "numb" to it by now), I can't really fault him for not realizing their feelings too much either. When Hachiman and Yukino were walking back together to the hotel after eating ramen with foreveralone, Yukino told him to keep away from her because she did not want to be seen together with him. Those kinds of things would make anyone think that they aren't liked, especially someone with tattered confidence like Hachiman.