I think the "them" reference was intentionally made to be vague. It makes it look like the Sumimura's had something to do with his death and probably led to Tokine's cold attitude to Yoshimori.

What Tokine's father was refering to could be either someone from the Urakai or even something we haven't heard about yet.