The most common response to a privacy concern in an LLM product is to update the system prompt.
Add a line. "Do not include patient names in the output." "Omit any personally identifiable information from third parties." Something like that. It feels like an obvious fix, it takes ten minutes to ship, and if anyone asks about your privacy controls, you can say you have them. This is what a lot of teams are actually doing right now.
A study published in Frontiers in Digital Health on March 31, 2026 found that it doesn't work reliably. The researchers tested three models, LLaMA, Mixtral, and Claude, generating clinical documentation from recorded patient-provider conversations. All three leaked third-party personal information in their outputs. Privacy instructions in the system prompt reduced leakage but didn't stop it.
That's worth sitting with for a second.
What the study actually tested
The setup mirrors how ambient documentation products actually work. A patient-provider conversation happens. The audio gets transcribed and fed to a model. The model generates a structured clinical note. The researchers wanted to know how often information that appeared in that conversation, specifically information about third parties who weren't the patient and weren't the provider, ended up in the note.
Third-party personal information matters because it's often incidentally disclosed in clinical conversations without anyone intending for it to enter the medical record. A patient mentions that her sister has a history of addiction, or that her father is in assisted living. That information about the sister, about the father, is now on tape. And when the LLM processes the conversation to write a note, it has access to all of it.
The researchers found that all three models, LLaMA, Mixtral, and Claude, incorporated this kind of third-party information into generated notes even without being asked to. The models were doing what they're designed to do: synthesize the conversation and produce a note that captures clinically relevant details. The problem is that "clinically relevant" is a judgment call, and these models are not making that call correctly when it comes to incidental disclosures.
The prompt instruction problem
Here's the part that should matter most to product teams: they tried the obvious fix and it didn't fully work.
The study tested giving models explicit privacy instructions in the system prompt. Something like: do not include personal information about individuals other than the patient in the note. The instructions reduced the frequency of leakage. They didn't eliminate it. Every model still leaked some third-party information when privacy instructions were the only control in place.
This is actually a known failure mode in LLM behavior. You can give a model a rule, and the model will follow it most of the time. But "most of the time" is not an acceptable compliance bar in a HIPAA-governed clinical context. If your system prompt says "don't include third-party PHI" and your model omits it 94% of the time, you have a system that leaks patient-adjacent information roughly once in every 17 encounters. Scale that across a health system doing 500 encounters a day and you're looking at a lot of compliance events before anyone notices a pattern.
Prompt-based mitigations fail in these cases partly because of how instruction-following actually works in current LLMs. The models are not running a rule-checker. They're predicting the next token based on context. The privacy instruction is one input among many, and when the conversation contains third-party details that look clinically relevant, the model's completion instincts can override the instruction. This isn't a bug that can be patched by rewording the rule.
What actually reduced leakage
The study did find controls that worked better. Two of them.
First, structured output formats. When the model was generating notes in a defined structure, specifically SOAP format (Subjective, Objective, Assessment, Plan), leakage rates dropped compared to free-form note generation. The structure constrains what goes where. If a model is generating the "Assessment" section, there's less natural space for incidental family history details. The format does some of the work that the instruction alone can't do.
Second, a two-stage generation approach. Instead of generating the final note in one pass, this approach generates a draft and then runs a second pass specifically to review and edit out information that shouldn't be there. The two-stage process outperformed single-pass generation with privacy instructions. The editing pass is doing explicit content review, which is a different cognitive operation than generating text while trying to remember a rule.
These aren't exotic architectural choices. SOAP notes are standard. A generate-then-review pipeline is a pattern that shows up in a bunch of LLM product architectures already. But they require intentional engineering. They're not something you get by adding a line to your system prompt.
What this means if you're buying or building an ambient scribe
The ambient documentation market includes Nuance DAX, Nabla, Abridge, Suki, and a growing number of EHR-native features from vendors like Epic. Most of them are generating structured notes. Many are using multi-stage processing pipelines. Some are doing post-generation review.
The question to ask any vendor is not "do you have privacy controls?" Every vendor will say yes. The question is where in the architecture those controls are implemented. If the honest answer is "we have instructions in our system prompt," that's a problem the Frontiers study is directly pointing at. If the answer is "we run a structured generation pipeline with a review pass and we've tested our leakage rates against a defined benchmark," that's a different answer.
There's no public benchmark for third-party PHI leakage in ambient documentation products right now. That's actually worth flagging as a gap. The study this post is based on is one of the first systematic evaluations of this specific failure mode. The fact that it's evaluating open-source and API models generically, not specific commercial products, means that buyers don't yet have product-level data to compare.
If you're on a product team building an ambient documentation feature in-house, the architectural recommendations from this study are fairly direct. Use structured output formats. Implement a review pass before the note is returned to the clinician. Test your leakage rates against a dataset of conversations that include incidental third-party disclosures, because that's the specific scenario where leakage happens. And don't count on your system prompt instructions to carry compliance weight on their own.
The closed-weight versus open-weight finding
One other detail worth flagging. The study found differences between closed-weight and open-weight models in the kind of leakage they produced. Closed-weight models, which have undergone more extensive safety training, tended to leak more generic information. Open-weight models like LLaMA and Mixtral leaked more specific, identifiable details.
This has obvious implications for teams that are evaluating self-hosted open-weight deployments as a cost-saving alternative to API-based models. The appeal of running your own LLaMA instance is real: you keep the data on-premises, you control the infrastructure, you reduce per-query costs. But the Frontiers findings suggest that the open-weight models may carry higher privacy risk in this specific failure mode, which complicates the calculus.
Honestly, I don't think this is a reason to rule out open-weight models categorically. The study was testing base-level instruction-following under specific conditions. A self-hosted deployment with structured generation and a review pipeline might perform as well or better than a closed-weight API call without those controls. The architecture matters more than the model family. But teams that are comparing self-hosted and API options should be asking about leakage rates specifically, not just about general data handling.
The practical upshot
Clinical ambient documentation is a real and useful product category. The productivity evidence is solid, the clinician demand is there, and the workflow integration is maturing. I'm not arguing that teams should slow down or avoid the space.
But privacy controls in these products need to be treated as an engineering problem, not a prompt problem. The Frontiers study is a direct challenge to any team that has checked the "privacy mitigations" box by updating their system prompt. That check is not sufficient. The controls that actually work, structured output formats and multi-stage generation with explicit review, require intentional architecture decisions.
If you work at an ambient documentation vendor and you think the study's methodology doesn't reflect how your product is built, I'd genuinely like to see the data. This is an area where the field needs more product-level evidence, not less.
Source: Frontiers in Digital Health, March 31, 2026 | Nuance DAX Copilot | Nabla | Abridge | Suki