Almost every ambient AI scribe on the market today listens. It does not look. The microphone captures what was said in the room, a model turns that into a note, and the clinician edits. That architecture has been good enough to build a multi-billion-dollar documentation category on, but it carries an assumption that nobody states out loud: that everything clinically relevant in an encounter gets spoken.
A lot of it doesn't. The clinician reads a label. The patient holds up a pill bottle. Someone points at a blister pack and says "this one." A purely audio scribe hears "this one" and has no idea what "this" is.
A study published in npj Digital Medicine on February 26, 2026 by Bradley Menz, Ashley Hopkins, Michael Sorich and colleagues at Flinders University put a number on that gap. They built an AI scribe that could see, fed it medication-history interviews, and measured what changed. The headline number is the kind that should make product teams stop and re-read it. Omission errors fell from 358 with audio alone to 10 with video.
What they actually built
The setup was deliberately simple. Ten clinical pharmacists conducted 110 simulated medication-history interviews. Each pharmacist wore Ray-Ban Meta AI Wayfarer smart glasses, which captured both the conversation and the first-person visual field, so the recording included whatever the pharmacist was looking at: the medication packaging, the printed labels, the dosing instructions.
The recordings were processed by Google's Gemini-Pro-2.5, a multimodal model that can take both audio and video as input. Ten recordings were used to iterate on the prompt. The remaining 100 became the test set, evaluated across 2,160 individual data points spanning patient details and medication-specific fields like drug name, dose, frequency, and indication.
Medication reconciliation is a smart task to test this on. It's high-volume, it's error-prone, and the errors have teeth. Getting a dose or a drug name wrong in a medication list isn't a documentation inconvenience, it's a patient-safety event. And critically, a lot of the relevant information lives on the physical packaging rather than in the spoken exchange.
The audio-versus-video result
When the scribe had video, it hit 98% accuracy across all 2,160 data points, with field-level accuracy running from 96% for patient details up to 99% for dosing directions and indication.
When the same pipeline was restricted to audio only, accuracy dropped to 81% (P < 0.001).
The mechanism behind that 17-point gap is the interesting part. Of the 46 total errors the video scribe made, 36 were commission errors (it wrote something wrong, like a misheard medication name or a wrong date of birth) and only 10 were omissions (it left something out). The audio-only condition produced 358 omission errors. The visual channel wasn't marginally improving transcription quality. It was filling in entire fields that the audio simply never contained, because the information was on a label the pharmacist read silently rather than something anyone said aloud.
This is the part I'd want every team building ambient documentation to internalize. The dominant failure mode of audio-only scribing in this task wasn't mishearing. It was not-hearing. You can tune an audio model forever and you will not recover information that was never spoken.
Why this reframes the scribe problem
The ambient-scribe market has mostly been competing on transcription fidelity, note structure, and EHR integration. Those matter. But this study suggests the ceiling on audio-only scribing has little to do with transcription quality. What caps it is how much of the clinically relevant encounter is spoken aloud in the first place.
That fraction varies enormously by specialty and task. A psychiatry session is almost entirely spoken, so an audio scribe captures nearly everything. A medication reconciliation, a wound check, a dermatology visit, a physical exam where the clinician palpates and observes without narrating every finding, these are encounters where a meaningful share of the signal is visual or tactile and never makes it into the audio stream. An audio-only scribe in those settings isn't 90% of the way there with some rough edges. It's structurally blind to a category of information.
The Flinders group is careful about scope, and so am I. This was a simulation. Australian pharmacists, English-speaking standardized patients, good lighting, freshly printed labels, medication packages present and visible. The authors list those constraints plainly. Real clinics have bad lighting, overlapping speech, patients who left their pill bottles at home, and packaging in a dozen languages. The 98% number will not survive contact with that environment unchanged.
But the audio-versus-video comparison holds up against most of those caveats, because both conditions ran on the same recordings. Whatever degraded the video score in a messier setting would degrade the audio score too. The gap is the finding, and the gap is large.
The product questions this raises
If you're building or buying ambient documentation, a few things follow from this that weren't obvious before.
Audio-only is a deliberate design constraint, not a neutral default. Most teams chose audio because it was the tractable input, not because they decided visual data didn't matter. That's a defensible engineering call, but it should be a stated one, with a clear view of which encounter types it underserves. A scribe that's excellent for telehealth talk-therapy and silently lossy for in-person med rec is two different products wearing one badge.
The hardware question just got real. Smart glasses in a clinical setting sound futuristic until you remember that the alternative, audio-only capture, is leaving documented errors on the table in exactly the tasks where errors hurt most. I don't think this study means everyone straps on Ray-Bans next quarter. I do think it means the "what does the device capture" decision moves from an afterthought to a design axis you have to reason about explicitly.
Commission errors don't go away, they become the main event. Once you solve omissions with a visual channel, 36 of 46 remaining errors were the model confidently writing the wrong thing. Misread medication names, wrong dates of birth. Those are arguably more dangerous than omissions, because an omission is a blank a clinician might notice and a commission is a plausible-looking falsehood they might not. A vision-enabled scribe doesn't reduce the need for clinician review. It changes what the reviewer should be looking for.
Evaluation has to test the visual cases on purpose. If your eval set is built from talk-heavy encounters, an audio scribe and a vision scribe will look nearly identical, and you'll conclude the camera isn't worth the trouble. The whole point of this study is that the value of vision only shows up when you deliberately include tasks where information lives off the audio channel. Buyers running pilots should weight their test cases toward med rec, exams, and procedural documentation, not against them.
What I'd watch next
The multimodal frontier models needed for this (Gemini-Pro-2.5 here, and the comparable releases from the other labs) are now capable enough that the model is no longer the bottleneck. The hard parts left are capture, consent, and workflow. Recording video in an exam room raises privacy and patient-consent questions that audio recording already raised, only sharper, and the authors flag that they haven't solved it. There's a real chance the regulatory and consent overhead slows visual scribing more than the technology does.
Still, the direction this points is hard to argue with. The ambient scribe category was built on the premise that listening to the room is enough. This study is a clean, quantified demonstration that for a large class of clinical work, it isn't. Three hundred fifty-eight omissions becoming ten says the input was wrong, not that the model needed a little tuning.
Paper reference: Menz, B. D., Hopkins, A. M., Sorich, M. J., et al. (2026). Vision-Enabled AI scribes reduce omissions in clinical conversations: evidence from simulated medication histories. npj Digital Medicine, 9, Article 287. https://www.nature.com/articles/s41746-026-02494-9 | DOI: 10.1038/s41746-026-02494-9
Related: Barriers and opportunities of scaling ambient AI scribes for clinical documentation across diverse healthcare settings. npj Digital Medicine (2026). https://www.nature.com/articles/s41746-026-02554-0
Related: An evaluation framework for ambient digital scribing tools in clinical applications (the SCRIBE framework). npj Digital Medicine (2025). https://www.nature.com/articles/s41746-025-01622-1