Technology to convert speech into sign language – and sign language into speech – has been launched by New York University Abu Dhabi.
The university has created a company, ChatSign, to promote the system, which works with Arabic or English, and Emirati sign language and US sign language.
Those behind the technology say that it offers a more cost-effective option than hiring a human translator and, unlike a person, is always available.
The hope is that the technology could be deployed everywhere from shopping malls to restaurants, government departments and universities.
While there are other AI-based sign language systems available, ChatSign says its technology stands out from competitors by offering bidirectional translation in real time.
There could be a large market for the technology, said Zhifei Maggie Li, ChatSign’s chief executive and co-founder, who completes her undergraduate mathematics and computer science degree at NYU Abu Dhabi this week.
World Health Organisation figures indicate, she said, that 430 million people worldwide are affected by deafness and disabling hearing loss, around a quarter of whom are signers.
“That’s one per cent of the population that needs translation between sign language and speech to be included into society,” she told The National.
“Even if they can find a human interpreter, it’s going to be super expensive, they’re not going to be available 24/7, and not that many interpreters exist in the market.
“We thought if our AI can be solving this problem, why not have it? We do not want to replace human interpreters, but assist people and fill the gap we find in the interpreter market.”
Communication is key
The patented technology has also been developed by Dr Yi Fang, ChatSign’s founder who is an NYU Abu Dhabi associate professor and director of the university’s embodied AI and robotics lab, and Zhenhua Li, ChatSign’s chief product officer and a co-founder.
It can convert Arabic or English into Emirati sign language or American sign language, and vice versa. It also works with text. Other languages will be added, with Chinese and German probably next on the list.
“The idea is [to convert] any language to any sign language without any difficulty. We are not restricted. It’s the end users’ choice,” Dr Fang told The National.
“Our devices are going to provide a very user-friendly interface to help the users select an input for their needs and their choice.”
The sign language is produced by an avatar, a humanlike figure on a screen.
ChatSign is in discussions with users in many settings, including health care, government, retail (such as shopping malls), food and drink (including restaurants), transport (such as airports) and education (such as universities).
The company has created its own hardware, which includes a screen showing somebody performing sign language that can be used at, for example, conferences, to sign what is being said.
Expanding possibilities
They also have a compact system for one-to-one interactions, such as at a help desk in a shopping mall. This system has a camera for the sign language input, a microphone for the spoken language input, a screen display for the sign language output, and a speaker for the spoken language output.
ChatSign says its system can operate offline without internet access. Further innovations are in development, including glasses that display sign language on the lenses.
The company was founded in 2024 and rapidly developed prototypes, which were used on campus late last year to produce a translation at an event with 200 students.
The technology has been presented at events including conferences, and the full-body screen will be showcased at the NYU Abu Dhabi commencement ceremony on Wednesday.
“It’s going to be, in real time, translating all of the announcements, the speakers,” Dr Fang said.
“It’s going to be installed in a people of determination area where we’re going to invite deaf and hard-of-hearing guests sit and enjoy engaging in our commencement.”
NYU Abu Dhabi’s interim vice chancellor, Fabio Piano, said that as a commercial AI technology spin-out company, ChatSign represented “an important milestone” for the university.
“It reflects the university’s broader mission to translate research and discovery into ventures that address real societal needs,” Mr Piano said in a statement.
