LLMs Are Strong at Naming Diagnoses. They're Still Weak at the Harder Part.
There's a version of the clinical AI story that gets told mostly in press releases and funding announcements. In that version, LLMs are approaching physician-level diagnostic performance, the benchmarks are improving every quarter, and the remaining gap is mostly a matter of regulatory process rather than capability.
A study from Mass General Brigham tells a more specific and more useful story. It's not that LLMs can't diagnose. It's that the task they're good at is not the task clinical diagnosis actually requires.
Understanding that distinction is more valuable for health-tech product teams than knowing which model scored highest on which benchmark.
What PrIME-LLM Measures and Why It Matters
Most prior evaluations of LLMs in clinical settings used closed-ended formats: present the model with a complete patient case, ask it to select from a list of possible diagnoses. That's a recall task. It measures whether the model can recognize a clinical pattern when all the relevant information is already assembled.
The Mass General Brigham team used a different methodology called PrIME-LLM, for Progressive Iterative Medical Evaluation. The distinction is structural. Rather than presenting a complete case and asking for a final diagnosis, PrIME-LLM presents clinical information sequentially, the way it actually becomes available during a patient encounter. The model receives the chief complaint first. Then the history. Then the exam findings. Then the initial labs. At each stage, it's asked to generate a differential diagnosis and recommend next steps, with incomplete information.
That's closer to what clinical reasoning actually requires. A physician seeing a patient for the first time doesn't receive a complete dataset and then pattern-match to a diagnosis. They receive a chief complaint, form an initial differential, decide what information to gather next, update their estimate as new data comes in, and commit to a plan under uncertainty. The iterative, sequential nature of that process is what PrIME-LLM is designed to stress-test.
The researchers evaluated 21 LLMs and found scores ranging from 64% (Gemini 1.5 Flash) to 78% (the top performers, including Grok 4 and GPT-5). The conclusion was direct: "off-the-shelf large language models are not ready for unsupervised clinical-grade deployment."
Sixty-four to seventy-eight percent sounds competitive if you're used to reading benchmark comparisons. On a clinical reasoning task, it means roughly one in four to one in three cases where the model's iterative reasoning is wrong. That's not a rounding error. In a high-stakes deployment context without human oversight, it's a patient safety problem.
The Closed-Ended Versus Open-Ended Distinction
A study published in Science in April 2026 found that OpenAI's o1 model outperformed attending physicians on structured ER diagnostic tasks, scoring 67% on real ED records against physician scores of 50% and 55%. I wrote about that finding in detail here.
Both the MGB and Science studies are testing clinical reasoning. The differences in their findings aren't contradictory. They're measuring different things, and the differences tell you something important.
The Science study used text records from real ER visits and asked models to identify the final diagnosis. That's closer to a closed-ended task: the information is assembled, the case is defined, the question is whether the model can name the right answer. On that task, a reasoning-optimized LLM in 2026 performs comparably to or better than a non-specialist physician working from the same text.
The MGB study tested the same kind of model on the sequential, uncertainty-laden version of the same work: given partial information, what do you think is going on, what should we check next, and how does your thinking change as data comes in? On that task, the same class of models is substantially weaker.
For product design purposes, this distinction isn't academic. It maps directly to where in a clinical workflow an AI decision support tool can carry its own weight versus where it needs a human in the loop.
The "Off-the-Shelf" Qualifier Is Load-Bearing
The MGB conclusion is specifically about off-the-shelf LLMs. That phrase is doing real work.
General-purpose foundation models are trained on broad internet-scale data with some degree of medical content. They know a lot about medicine because medicine is well-represented in the text they've seen. But they aren't trained specifically on clinical diagnostic tasks, clinical dialogue, or the sequential reasoning patterns that characterize real-case workups.
Fine-tuned clinical models tell a different story. Google's Med-PaLM 2 showed substantially stronger performance on structured clinical question-answering than the base PaLM 2. Microsoft's BioGPT was trained specifically on biomedical literature. Abridge's model is fine-tuned on clinical encounter data. None of these have been evaluated on PrIME-LLM, so a direct comparison isn't possible, but the principle is consistent: models trained on clinical reasoning tasks perform better at clinical reasoning tasks than models trained on general text.
The MGB finding is a ceiling on what you can expect from dropping an off-the-shelf frontier model into a clinical workflow without further adaptation. It's not a ceiling on what purpose-built clinical AI can do. Product teams who read this study as "AI can't do clinical reasoning" are drawing the wrong conclusion. The right conclusion is "general-purpose LLMs can't do this reliably unsupervised, and fine-tuning for the specific task matters."
Where the Human-in-the-Loop Checkpoint Belongs
Here's where the research has direct product design implications.
The standard instinct in health-tech product design is to put the human checkpoint at the end. The AI does the work; a clinician reviews the output before it becomes an action. That makes intuitive sense for many AI tasks, and it's not wrong in all clinical contexts.
But the PrIME-LLM finding suggests the checkpoint architecture needs to be different for iterative diagnostic reasoning. The model is weakest at the beginning of a case, when information is incomplete and the differential is wide. It's strongest when the full picture is assembled. A review-at-the-end design puts human oversight where the model is most reliable and leaves the model unsupervised where it's most likely to go wrong.
A better architecture for iterative clinical AI tools places human-in-the-loop checkpoints at the early iterative stages. Not just a final sign-off on the output, but a physician reviewing and correcting the differential at case onset, before the model has propagated an early reasoning error through several subsequent steps.
This is operationally harder. It requires more physician touchpoints, not fewer, in the phase of the workflow where the efficiency argument for AI is supposed to be strongest. But the MGB finding is a reason to take that tradeoff seriously rather than designing around it.
The clinical AI products that will hold up to regulatory and liability scrutiny over the next few years are the ones that can show where their human oversight is, not just that human oversight technically exists somewhere in the process. "A physician reviews the final recommendation" is weaker than "a physician reviews and approves each iterative diagnostic update before the system proceeds." The second architecture is harder to build and harder to sell, but it's what the evidence says the iterative reasoning gap actually requires.
What a Responsible Prospective Trial Would Need to Show
The MGB study is a benchmark evaluation, not a clinical deployment study. It tells you how the models perform on a structured test. That's necessary evidence. It's not sufficient evidence for what the models do in a real clinical environment with real patients.
A prospective trial would need to do several things the benchmark can't. It would need to compare AI-assisted diagnosis against physician-only diagnosis on the same patient population, with patient outcome data collected at meaningful follow-up intervals. It would need to specify the exact deployment architecture: which model, with what fine-tuning, at which workflow touchpoints, with what human override structures. It would need to measure not just whether the final diagnosis was correct, but whether the diagnostic process itself was efficient and safe, including time to diagnosis, unnecessary tests ordered, and adverse events attributable to diagnostic delay or error.
It would also need to run long enough and across diverse enough patient populations to capture performance on edge cases. The MGB finding that models struggle at case onset with incomplete information is most dangerous for complex cases that don't fit standard templates. Those cases are rare in any given patient panel, but they're the cases where errors have the most serious consequences.
The field doesn't have that prospective trial yet for iterative diagnostic AI. The standard in most clinical AI regulation is retrospective validation, which tests the model on historical cases. That's better than no validation. It's not the same as a randomized comparison of patient outcomes.
A product team seeking to build an iterative diagnostic AI tool responsibly should treat the prospective trial design as a product requirement, not a post-launch research question. Defining the trial before deployment means defining the human oversight architecture, the data collection plan, and the success criteria in advance, rather than after the first adverse event.
The Practical Takeaway
The MGB study doesn't tell you LLMs are useless in clinical settings. It tells you something more specific: off-the-shelf general models have a performance gap on iterative reasoning under uncertainty that is large enough to make unsupervised deployment a patient safety problem.
That gap has a shape. It's worst at case onset with incomplete information. It closes as information accumulates and the case becomes more defined. The product design implication is to match your oversight architecture to the shape of the gap, not to treat oversight as a uniform feature that exists somewhere in the workflow.
Fine-tuning for the specific clinical task matters. Checkpoint placement within the iterative process matters. And the distinction between "the model named the right final diagnosis" and "the model reasoned well through an uncertain, unfolding case" is the distinction that will separate the clinical AI products that survive long-term scrutiny from the ones that don't.
The 64 to 78 percent range is the number to remember from this study. One in four to one in three cases where iterative reasoning goes wrong is not a number that survives contact with a patient safety committee. Building around it rather than acknowledging it is how products get recalled.
Sources: Mass General Brigham press release | OpenAI o1 ER diagnosis study in Science | Google Med-PaLM 2 | Microsoft BioGPT | Abridge