Even Sylphie doesn't know about it, I believe. I suppose it wouldn't be overly charming in the new wife's eyes to learn the husband worships another woman's underwear.
I imagine if he explained it as "it's the only physical momento I have of the person that taught me magic as a child", she'd probably wave it off as one of his quirks.