"One death is a death too many. Plain and simple," Hassan Al Thawadi, chief executive of Qatar's World Cup organising committee, told British TV show Piers Morgan Uncensored.
Mr Al Thawadi said there had been three work-related deaths and 37 non-work-related deaths at the stadiums.
When asked what the exact number of migrant workers who died in construction for the World Cup in the past 12 years, he said: "The estimate is around 400. Between 400 and 500."
"Every year the health and safety standards on the sites are improving, at least on our sites, the World Cup sites," Mr Al Thawadi said.
He said German and Swiss trade unions had commended the standards.
"The improvements that happened aren't because of the World Cup, they are improvements we knew we had to do because of our own values."
Mr Al Thawadi said the tournament served as a "catalyst" to speed up the process of improving health and safety and accommodation standards.
Qatar relies heavily on migrant workers, who make up about 95 per cent of its labour force. It has enacted labour reforms in recent years that have been welcomed by the UN and rights groups.
But those groups say exploitation is still widespread and more should be done to protect workers from wage theft and harsh conditions.
Mr Al Thawadi said he was "proud" of the tournament so far and that the first World Cup in the Arab world was producing some exciting stories on and off the pitch.
He said Arab fans were supporting each other.
"For our region, as Qataris, as Arabs, we're extremely proud," he told Piers Morgan.
But Mr Al Thawadi said he felt BBC coverage of the tournament was unfair, including a decision to have commentator Gary Lineker talk about Qatar's human rights record during the opening ceremony.
He said Qatar had invited Lineker to meet him and visit Qatar in February.