It could be. We were never told why he retired, but I'm totally expecting something like that. Maybe he pissed off some actress or something and the social circle banished him. Those flashbacks give me that impression, amongst others. It easily could have been that his narcissism became unbearable. "I'm too good for 30 yen" tells a similar story.