That makes it sound like he shouldn't be equally concerned about the loss of his son.
His WHOLE family was lost, not just the ones he fucks. He FINALLY finds one of them, and instead of being happy about finding his son, he's pissed cause his son didn't find everyone else for him?
Seems to me like he's pissed at himself for his own failure, and projecting that onto Rudy.