An AI Model Out-Diagnosed ER Doctors in a Real Hospital Study. The Fine Print Matters.

On April 30, 2026, a paper in Science got the kind of coverage that tends to generate more heat than light. Harvard Medical School researchers found that OpenAI's o1 reasoning model outperformed attending physicians at diagnosing patients across a series of emergency medicine tasks. The headlines ran predictably: AI beats doctors. Medicine is next. The robots are coming.

None of that framing is what the paper actually says. But what the paper actually says is worth reading carefully, because it's the most rigorous published test of LLM clinical reasoning to date, and the results are genuinely interesting, even after you account for all the caveats.

What the Study Measured

The research came out of Harvard Medical School and Beth Israel Deaconess Medical Center, led by Arjun Manrai and Adam Rodman. It's not a laboratory evaluation using medical board questions. The team tested o1-preview on six categories of clinical reasoning tasks, including differential diagnosis, diagnostic testing, probability estimation, and treatment planning, using real electronic health records from emergency department patients.

In the real-world ED component, the model correctly identified the exact or near diagnosis in 67% of triage cases. Two internal medicine attending physicians, given identical patient information from the same records, scored 55% and 50% respectively. On five complex expert-developed scenarios, o1 received a median score of 89%. Physicians using conventional tools, including Google searches and UpToDate, scored 34% on the same scenarios.

Those are large gaps. They're also gaps on a constrained task, evaluated in controlled conditions, which is not the same thing as being a better doctor.

The STAT News writeup quotes Rodman directly: "We're already at the ceiling" of what current benchmarks can measure. The study's framework is a more sophisticated version of the clinical reasoning evaluations researchers have used since the 1960s. It's also still a framework, not a hospital.

Where the Methodology Gets Complicated

The most pointed criticism came from emergency physician Kristen Panthagani, who noted that the physician comparison group was wrong for the task. Gizmodo reported her observation: the study compared AI to internal medicine physicians, not emergency medicine physicians, on emergency department cases. That's a meaningful distinction. Internal medicine clinicians rotating through the ED bring different pattern recognition than emergency medicine attendings who spend their entire career triaging that case mix. If the comparison group is systematically suboptimal for the task, the performance gap between AI and physician narrows, possibly significantly.

The study also used text only. The model received chart notes, lab values, and clinical documentation, which is what's in the electronic health record. Real diagnostic reasoning involves imaging, physical exam findings, the way a patient looks and moves and speaks, how a family member in the room is behaving, and dozens of other signals that don't make it into a text file. The model didn't have access to any of that.

TechCrunch's coverage noted the hallucination concern: co-author Peter Brodeur acknowledged the model "might get the top diagnosis right but also suggest unnecessary testing that could expose a patient to harm." o1-preview does hallucinate, by the authors' own admission. In a clinical context, a confident wrong answer is worse than a confident right answer, because it can close off the diagnostic search before it should stop.

Finally, the model tested is already outdated. The research was originally posted as a preprint in December 2024 and uses o1-preview, which OpenAI has since superseded with o1, o1-pro, o3, and subsequent iterations. The paper was published in April 2026, nearly 17 months after the underlying work was run. In a field moving as fast as foundation models, that's a long lag.

Why the Result Still Matters

Setting aside the caveats, the study is the most credible published evidence that reasoning-optimized LLMs have crossed a threshold on structured clinical tasks. That's not nothing.

Prior evaluations of LLMs in clinical settings mostly used multiple-choice medical licensing questions, which test recall more than reasoning. The Harvard team's framework tests something closer to actual clinical thinking: given incomplete information, generate a plausible differential, choose an appropriate next test, update your estimate, and commit to a management plan. That's harder, and a 67% accuracy rate on messy real-world ED records is a different kind of finding than a 90% score on USMLE questions.

The NPR piece quotes Rodman again on what the study is actually calling for: prospective trials. Evaluating an AI model on historical records, even real ones, is not the same as deploying it in a live clinical environment and measuring what happens to patients. The study is an argument for why those trials are now worth running, not an argument for why hospitals should hand diagnostic authority to an LLM.

What the Behavioral Science Literature Would Add

Clinical decision-making research has spent decades establishing that physicians don't make decisions in the way medical school trains them to. Cognitive load, time pressure, anchoring bias, availability heuristics, and diagnostic momentum all shape the actual reasoning process, often without the clinician being aware of it. Physicians make errors not because they lack knowledge but because the conditions of clinical work consistently erode the kind of deliberate, sequential reasoning that multiple-choice exams measure.

An AI model operating on text records doesn't experience those same pressures. It doesn't have twelve other patients waiting. It doesn't just saw off a 12-hour shift before looking at the chart. It doesn't anchor on the first diagnosis the referring physician suggested. Whether that means it's actually better at diagnosis, or whether it's just operating under conditions that no real clinician ever faces, is a question the study can't resolve from the outside.

The University of Sheffield's Wei Xing, quoted in coverage of the study, raised an equity concern that the paper doesn't address: the model's performance across patient subgroups, including older patients, non-English speakers, and patients from lower-income backgrounds, wasn't reported. If the model works better for some patients than others, that's a clinically significant finding, and its absence from the analysis is a gap.

The Accountability Question

Rodman was direct on this point in multiple interviews: there's no formal accountability framework for AI diagnoses. If an attending physician makes a diagnostic error, there are legal, professional, and institutional mechanisms for addressing it. If an AI model makes a diagnostic error and a physician followed its recommendation, the accountability chain is genuinely unclear.

That's not a reason to stop studying LLMs in clinical settings. It's a reason to be cautious about deployment timelines and deployment contexts. Advisory use, where a clinician is explicitly using the model as a second opinion on a specific narrow question, looks different from workflow integration, where the model's output is the default recommendation a physician sees first.

The Harvard Magazine writeup notes that 81% of US physicians now report using AI professionally, up from 38% in 2023. That adoption is happening regardless of whether the prospective trials exist. The clinical AI field's credibility problem, and the patient safety risk, is that deployment is outrunning evidence. The Harvard/Beth Israel study is evidence in the right direction. It's not enough evidence to justify the deployment patterns already underway.

The Honest Bottom Line

OpenAI's o1 model, on text-based diagnostic tasks using real ED records, outperformed a comparison group of internal medicine physicians on structured clinical reasoning. The effect size is large enough to take seriously. The comparison group, the text-only input constraint, the retrospective design, and the absent subgroup analysis all limit what you can actually conclude from that result.

What you can conclude: reasoning-optimized LLMs have crossed a threshold on text-based clinical reasoning tasks that prior models hadn't cleared. That's a sufficient result to justify the prospective trials the authors are calling for. It's not a sufficient result to conclude AI should be making or meaningfully shaping diagnosis decisions in real clinical settings without substantially more evidence.

The study is worth reading. The headlines were not worth believing.