Tokyo 2020 chief Yoshiro Mori bowed to pressure and resigned on Friday over sexist remarks.
Who would succeed him was not clear after opposition emerged to his favoured successor.
The resignation and the leadership vacuum left behind by the controversy add to the woes of organisers struggling to win over a sceptical public less than six months before the virus-delayed Olympic Games.
Mr Mori, 83, sparked domestic and international outrage by claiming last week that women speak too much in meetings, with officials, sports stars and sponsors calling his remarks inappropriate.
On Friday he announced he would step down with immediate effect.
"My inappropriate statement has caused a lot of chaos. I would like to express my sincere apologies," he told a meeting of Tokyo 2020's executive board and council called to discuss his remarks.
"I wish to resign as the president as of today," he said.
"What is important is to hold the Olympics from July. It must not be the case that my presence becomes an obstacle to that."
There was confusion about who would succeed Mr Mori, who had initially selected well-known sports administrator Saburo Kawabuchi, 84, to replace him.
The transition appeared a done deal, with the former footballer speaking to local media about his priorities in the new job.
But opposition to the selection of another octogenarian, and Mr Mori's control over the process, quickly mounted.
"[On] the organising committee there are some voicing concern," the Asahi Shimbun newspaper reported on Friday morning.
"I don't think an old man like him taking over will convince the public," it cited a source involved in organising the Games as saying.
Hashtags opposing Mr Kawabuchi's appointment trended on Twitter in Japan, and the country's Olympic Minister Seiko Hashimoto insisted "nothing has been decided".
"The organising committee will make a decision ... while listening to opinions from a range of people," she said.
"It is desirable to go through a formal procedure."
By early Friday afternoon, local media reported Tokyo 2020 had come under pressure from the government over the appointment and that sources close to Mr Kawabuchi said he had decided to turn down the job.
His withdrawal leaves the race for the key post wide open, although reports suggested Ms Hashimoto, a former winter and summer Olympian and one of just two women in the Cabinet, was a leading candidate.
Mr Mori's resignation caps just over a week of uproar after he told members of Japan's Olympic Committee that women have difficult speaking concisely, "which is annoying".
He apologised but then defended his remarks and said: "I don't speak to women much."
The comments drew fire at home and abroad. Several hundred Olympic volunteers have since withdrawn and a petition calling for action against him gathered nearly 150,000 signatures.
While Mr Kawabuchi was considered an able administrator, the appearance of Mr Mori hand-picking his successor did not go down well.
"It makes no sense for a resigning chief to appoint his successor," one Tokyo 2020 board member told the Mainichi Shimbun.
"There are steps to this process."
The fall-out over Mr Mori's remarks came with organisers already battling public doubt about holding the huge international event this summer.
About 80 per cent of Japanese polled in recent surveys back either further postponement or an outright cancellation.
Organisers tried to quell the disquiet by releasing virus rulebooks for athletes, officials and media, including restrictions on movement and regular testing.
But with Tokyo and other parts of the country under a virus state of emergency, doubts persist about the event's viability.
The first Olympic test event of the year was postponed because of Japan's strict virus entry rules.
Japan's first vaccine approval is expected over the weekend, with thousands of medical workers first in line to be inoculated, probably by the end of February.
But a wider campaign will move slowly, with vaccination of the elderly not expected to start until April.
