I wouldn't be so certain he's so dumb about females. He might not be a real career bureaucrat but he's still protecting himself by not messing with his boss - no matter what manner of a boss. I think the fact how he distances himself from her even when she's not present shows he's all the time trying to draw the line. Though I don't deny he's a somewhat stiff dude; more Japanese than French for sure..

Ryoko being a daughter of some mega corporation owner would mean in Japan, I believe, that normally she wouldn't certainly marry any noname small-time official. So, in that sense I reckon Izumida would be protecting himself from a broken heart as well because he doesn't especially look like a playboy.