Overview
Artificial intelligence is rapidly entering healthcare. Yet a critical new study reveals that generative AI models still fall short where it matters most — clinical reasoning. Researchers from the MESH Incubator at Mass General Brigham tested 21 large language models (LLMs) and found that these tools struggle badly with the early, open-ended stages of diagnosis. The findings, published in JAMA Network Open, raise urgent questions about AI’s readiness for unsupervised clinical use.
What the Study Tested
Simulating Real Clinical Scenarios
The research team asked 21 different LLMs to work through 29 published clinical cases. Crucially, they did not hand over all patient information at once. Instead, they fed the models data gradually — starting with basics like age, gender, and symptoms, then adding physical exam findings and lab results step by step. This approach closely mirrors how real clinical cases unfold in practice.
Medical student evaluators assessed each model’s performance at every stage. Their assessments then fed into an overall score for each AI system.
The AI Models Tested
The study covered a wide range of well-known AI systems, including the latest versions of ChatGPT, DeepSeek, Claude, Gemini, and Grok at the time of submission. By including so many models, the researchers ensured a broad and fair comparison across the current AI landscape.
How Researchers Measured AI Clinical Competency
Introducing the PrIME-LLM Framework
To go beyond simple accuracy scores, the team developed a new evaluation tool called PrIME-LLM. This framework assesses a model’s competency across four distinct stages of clinical reasoning:
- Generating potential diagnoses (the differential)
- Ordering appropriate diagnostic tests
- Arriving at a final diagnosis
- Planning treatment and management
Traditional evaluations often average performance across all tasks. That approach, however, can hide serious gaps. PrIME-LLM, by contrast, captures imbalances directly — so a model that excels at final diagnosis but fails at earlier reasoning steps receives a score that honestly reflects both strengths and weaknesses.
Key Findings: Where AI Succeeds and Fails
Strong at the Finish Line, Weak at the Start
The headline result is striking. All tested LLMs correctly reached the final diagnosis more than 90% of the time when given complete patient data. On the surface, this sounds impressive. Dig deeper, though, and the picture changes dramatically.
Every single model failed to produce an appropriate differential diagnosis more than 80% of the time. In other words, while AI can name the answer at the end, it routinely stumbles through the critical thinking that leads there.
Lead author Arya Rao, an MD-PhD student at Harvard Medical School and MESH researcher, put it plainly: “These models are great at naming a final diagnosis once the data is complete, but they struggle at the open-ended start of a case, when there isn’t much information.”
Performance Improves With More Data
On a more positive note, most LLMs performed better when researchers added laboratory results and imaging to the text-based information. Moreover, newer models generally outperformed older ones. This signals genuine, incremental progress — even if the gap between AI capability and clinical readiness remains large.
Why Differential Diagnosis Matters
The Heart of Clinical Medicine
A differential diagnosis is not a formality. It is the structured process by which a clinician lists all plausible explanations for a patient’s symptoms, then systematically rules them out. This step prevents dangerous diagnostic errors and drives appropriate testing decisions. It reflects the kind of reasoning that experienced physicians develop over years of practice.
Dr. Marc Succi, MD, executive director of the MESH Incubator and corresponding author on the study, explained the stakes clearly: “Differential diagnoses are central to clinical reasoning and underlie the ‘art of medicine’ that AI cannot currently replicate.”
Accordingly, an AI that can arrive at a correct final diagnosis but cannot construct a sound differential has not truly reasoned its way to that answer. It has, in effect, guessed well — but guessing is not clinical medicine.
What PrIME-LLM Scores Revealed
A Wide Range Across Models
The PrIME-LLM scores for the 21 tested models ranged considerably. At the lower end, Gemini 1.5 Flash scored 64%. At the higher end, Grok 4 and GPT-5 both reached 78%. While these numbers show measurable differences between models, none crossed the threshold that would indicate readiness for independent clinical deployment.
A New Benchmark for AI Developers
Beyond this study, PrIME-LLM offers something valuable to the broader industry. Dr. Succi described it as a standardized way to evaluate AI’s clinical competency — one that AI developers and hospital leaders can use to benchmark new technologies as they come to market. Rather than relying on marketing claims, healthcare decision-makers now have a rigorous, multi-stage evaluation framework to apply.
The Road Ahead for AI in Healthcare
Augmentation, Not Replacement
The study’s conclusions are unambiguous. Off-the-shelf LLMs are not ready for unsupervised clinical-grade deployment. Furthermore, their limitations are not merely technical — they reflect a fundamental gap between pattern recognition and genuine reasoning.
Nevertheless, the researchers are not dismissing AI’s potential. Dr. Succi emphasized that “the promise of AI in clinical medicine continues to lie in its potential to augment, not replace, physician reasoning, provided all the relevant data is available.” The key word here is augment. Used carefully, under close physician oversight, AI tools can still add value. Used autonomously, they introduce serious risk.
Therefore, healthcare institutions should treat current LLMs as decision-support tools — never as independent diagnosticians. As Succi put it, “large language models in healthcare continue to require a ‘human in the loop’ and very close oversight.”
