Wasn't it his aunt who told him that his mother hated him? Besides, it shows he wasn't insane back in the chunnin exam arc when he was babbling on to the sand inside his shell and calling it 'mother'.