Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) is conducting pioneering research to refine artificial intelligence so systems can understand Arab culture and navigate its complex regional dialects.
MBZUAI researchers have developed the first benchmark to measure AI models’ ability to understand and engage with Arab culture across 13 national dialects, revealing a striking gap between the models’ ability to understand what Arabs say and their ability to speak the way Arabs do in everyday life.
Lead researchers Fajri Koto, assistant professor in the Department of Natural Language Processing, and Muhammad Dehan, a researcher in the same department, told state news agency Wam that bridging this gap could pave the way for the next stage in the development of an Arabic-language AI.
Mr Dehan said Arabic there are more than 400 million Arabic speakers worldwide, but one of the main challenges in AI development is that models are trained almost exclusively on Modern Standard Arabic. As a result, they may appear fluent and capable in trials without having been sufficiently tested in natural conversations across the various dialects.
The researchers developed ArabCulture-Dialogue, which they describe as “the first benchmark for testing Arabic cultural reasoning in multi-turn conversations across Modern Standard Arabic and 13 national dialects from different Arab countries”.
The team recruited 26 native Arabic speakers in 13 countries to engage in conversation, covering 12 topics ranging from weddings and food to parenting, agriculture, arts and games.
“We gave the models three tasks: pick the culturally appropriate reply from a set of options; translate between Modern Standard Arabic and a specific dialect; and continue a conversation in a named dialect on request,” said Prof Koto.
“The strongest models did well at recognising the culturally appropriate answer … even when the conversation shifted from MSA to dialect. But when we asked them to produce dialect, for example, to translate a line into Emirati, or to continue a conversation in that dialect, performance dropped sharply.”
The study found that models performed better when dealing with customs shared across the Arab world, while country-specific customs proved more difficult. Emirati and North African dialect dialogues were among the most challenging of all.
Mr Dehan said: “The paradox is that the cultural knowledge is already present within the models but they sometimes require only a small amount of guidance.” Specifying the country and region associated with a conversation led to improved accuracy, he added.
The findings are of strategic importance to the UAE, which has made AI a national priority through the UAE National Strategy for Artificial Intelligence 2031 and the development of home-grown AI models, including Jais.
