Oh boy, there is so much fluff that its hard to pin down an exact chapter. Entire arcs are devoted to 'eliminating' single girls from the harem. That being said, the 'mystery' of the locket, which is the central story by my count, starts resolving around chapter 210. That is when the biggest bombs begin to drop and cross the point of no return. I hope thats helpful.