Patients have been warned against turning to artificial intelligence for at-home medical diagnosis after some AI systems failed to recognise crucial symptoms.
Researchers at Carnegie Mellon University's School of Computer Science in the US investigated how large language models such as GPT-5, Gemini and Claude responded to basic medical questions aimed at diagnosing illness.
The analysis showed the large language models (LLMs) occasionally invented false information when asked about details a doctor would routinely refer to when assessing a patient.
As part of the study, Siddharth Vohra, a robotics institute master’s student, asked popular LLMs to describe a medical image that had been intentionally omitted from the query.
But rather than requesting the missing image, in 18 per cent of the cases, the models fabricated a diagnosis based on the user’s age, gender and race.
Misleading
The results suggested almost one in five diagnoses presented by AI could be false or misleading.
Computer programmes also made assumptions that some patients had cancer, without referring to a telltale image that could help rule out the disease.
“A 65-year-old white man asking Claude about a skin mole receives melanoma in nearly every response,” said Mr Vohra, the study's author.
“When chest X-ray questions are presented, OpenAI’s GPT-5 names sarcoidosis for roughly 77 per cent of young black patients.”
Cancer specialists estimate that fewer than one in 10,000 moles become cancerous. Sarcoidosis – an inflammatory disease usually found in the lungs – is rare and appears in 10 to 60 of every 100,000 people, with some regional variations.
Mr Vohra said a diagnosis can change when only race, age, or gender is written into the prompt.
The research highlights flaws in the methods of self-diagnosis in the age of Dr Google and advanced AI.
By inputting symptoms, patients are bypassing professional medical opinion and placing their health in the hands of a computer programme instead.
While AI responses can look convincing and authoritative, they are not grounded in fact, and just small changes in the words used can drastically influence a model’s conclusions.
During the study, Mr Vohra focused on chest X-rays, brain MRI scans and dermatology images in 12 simulated patient profiles.
The AI models delivered 11,700 responses across Claude, GPT-5 and Google’s Gemini. The findings reinforced a major concern in the AI industry that users often assume models understand more than they actually do.
“About 82 per cent of the time, the models refused to provide a response because no image was attached,” Mr Vohra said. “However, in the remaining 18 per cent, the models invented a diagnosis instead of asking for the missing image.”
“There’s a general perception that AI models are smart because they perform well on certain tasks. But being good at one thing doesn’t mean they’re good at everything.”
Convincing diagnosis
According to the 2026 Edelman Trust Barometer Global Report, out of 1,000 people asked in the UAE, 59 per cent used AI to manage their health. The report also found that seven in 10 people felt confident in their ability to find answers to health questions and to make informed decisions.
“Large language models reached hospitals before the safeguards did,” said Ahmed Ashoor, chief technology officer for Outworks, a UAE technology service provider.
“A model produces a fluent answer whether or not the answer is true, and in medicine a confident wrong answer does more damage than no answer at all.
“Developers train most models on western populations, so accuracy drops for regional patients, from genetic risk factors to symptoms a patient describes in Arabic.”
Data privacy is a further issue holding back the effectiveness of LLMs in healthcare, Mr Ashoor said. “Health records rank among the most sensitive data a nation holds, and no ministry can hand them to systems outside its control,” he said.
“Accountability closes the list. When an algorithm shapes a clinical decision, someone must own the outcome, and many health systems cannot yet say who that is.”
Hospitals
Radiology is proving to be the most effective area of healthcare where AI is deployed.
Models flag abnormalities in scans with an accuracy close to that of a specialist, which gives every image a second read without adding a minute to the radiologist's day.
Hospital systems based on AI that predicts admissions, manages bed capacity and flags a deteriorating patient hours earlier also save lives without ever writing a prescription.
Nalla Karunanithy, chief executive of Digital Health and Omnichannel at Aster DM Healthcare, said the group’s hospitals use AI algorithms to help radiologists prioritise urgent cases and detect subtle findings faster.
Chatbots, however, were not part of hospital operations but were helpful in summarising medical literature, he said.
“Where large language models genuinely add value today is around the clinical encounter rather than inside the diagnostic decision itself,” said Mr Karunanithy.
“General models can carry biases from their training data, including around ethnicity and race, and can present incorrect answers with complete confidence.
“In a clinical setting, that confidence without accountability is not something that is encouraged at this point.
“Any AI that touches a diagnostic pathway at Aster works under physician oversight, and the physician remains responsible for the decision.”
