Most health-tech teams picking an AI layer right now are running the same evaluation. They spin up side-by-side tests across GPT-4.1, Claude Sonnet 4.6, Gemini 2.5 Pro, and maybe a Mistral or Llama variant for the on-prem conversation. They feed the models a clinical case set, score the outputs against a reference answer, and pick the one with the highest accuracy number. The deck that goes to the board has a chart with four bars and a circle around the tallest one.

That evaluation is mostly theater. The accuracy gap between the top three frontier models on clinical reasoning has been inside the noise for about eighteen months. MedQA scores, USMLE-style benchmarks, MultiMedQA, none of them separate the leaders by anything that matters to a real product. A two-point difference on a benchmark does not predict whether the tool will work for the 58-year-old Medicaid patient who opens the app at 11pm on a Tuesday.

The more useful review is harder to run and almost nobody is running it. Here is what it looks like.

Accuracy is a false north star

The framing every vendor pitch relies on is that the model is the product. If accuracy goes up, outcomes go up. This has never been true in health technology and it is not becoming true now. A 91% accurate clinical-reasoning model that produces a plan the patient does not follow is worth less than an 86% accurate model that produces a plan the patient actually does.

Behavior is the bottleneck, which means the review has to measure behavior-adjacent properties. Five questions matter more than the benchmark score.

1. Latency under real clinical workflow

The published numbers are a fiction. OpenAI says GPT-4.1 responds in 1.2 seconds at the median. Anthropic publishes similar numbers for Claude Sonnet 4.6. Those figures are measured on short prompts with no tool use, no retrieval, no structured output parsing, and no safety filtering on top.

Real patient-facing workflows look nothing like that. A triage chat with retrieval over a patient's chart, a structured FHIR output, and a safety reviewer running in parallel puts end-to-end latency somewhere between 4 and 14 seconds depending on context length and time of day. That is the number to test. A patient waiting 11 seconds for a response after describing chest tightness is a product problem the benchmark does not catch.

Measure p50 and p95 latency at your actual prompt length, your actual tool chain, and during the hours your users actually use the product. Run it for two weeks, not an afternoon.

2. Refusal patterns on edge cases

Every frontier model refuses differently. Claude tends to over-refuse on anything adjacent to self-harm or medication dosing, often adding a long preamble before answering. GPT-4.1 is more willing to engage but sometimes refuses to produce specific numbers a clinician actually needs. Gemini 2.5 has a habit of refusing mid-stream, which produces a worse UX than refusing at the start.

For a patient-facing product, the interesting question is not "does the model refuse dangerous content" (all of them do, most of the time). It is "what does the refusal feel like to someone in distress." A patient describing suicidal ideation to a diabetes coach deserves a specific, warm, useful response that routes them to care. What most models produce by default is a liability-shaped paragraph that reads like the patient just triggered a compliance filter.

Build a red-team set of 40 to 80 edge-case prompts drawn from your actual support tickets. Run them through every candidate model. Read the outputs. Half the decision lives there.

3. Bias patterns on clinical populations

The accuracy benchmark averages over a population that does not exist in any real clinic. Performance on the same task can swing 8 to 12 points depending on whether the patient is described as a 45-year-old white man or a 62-year-old Black woman with Medicaid coverage.

This is not theoretical. A 2024 study from Stanford showed meaningful differences in pain-management recommendations across race and insurance-status descriptors for every major model. Those differences have narrowed in the last year but they have not disappeared. If the product serves a population that is majority Medicare, majority non-English-first-language, majority rural, or majority low-income, the averaged benchmark is lying about what the product will do in the field.

Run the evaluation set twice. Once with neutral demographics. Once with the demographics your users actually have. If the gap is more than 3 points on anything clinically significant, the model choice is wrong or the prompting is wrong, and you need to find out which before launch.

4. Observability and incident response

Most AI providers treat observability as an afterthought. Anthropic's usage console is fine for cost tracking and useless for debugging a specific bad output. OpenAI's is slightly better. Azure OpenAI gives you the most control if you are willing to host it yourself. Bedrock sits in the middle.

For a product where a bad output can harm a patient, the question is not whether the model is accurate on average. It is how fast the team can see, reproduce, and fix a specific incident. Can engineers pull the exact prompt, retrieved context, system message, and output for a complaint filed three weeks ago? In most of the setups we have reviewed, the answer is no, because nobody built that pipeline before launch and logs were dropped at 14 days.

Before picking a model, pick the observability stack. If the vendor's native tooling is insufficient, budget for Langfuse, Helicone, or a custom logging layer. The model choice matters less than the ability to audit it after something goes wrong.

5. Cost at real scale

Vendor pricing decks assume linear scaling. The reality is that unit costs change shape around three inflection points: input caching (cuts cost 70 to 90% on repeated context), batch processing (cuts cost 50% on non-realtime flows), and volume negotiation (kicks in somewhere between 50M and 200M tokens per month depending on the vendor).

A health product running retrieval over a patient chart on every conversation can spend $0.40 per patient interaction on GPT-4.1 at list price. With input caching and batch for the non-synchronous parts, the same workflow runs closer to $0.08. At 50,000 daily active users, that difference is $6M per year.

None of that shows up in the accuracy chart. It shows up in the financial model six months after launch when someone notices the COGS per engaged user does not support the pricing tier.

A concrete checklist

Any team picking an AI layer for a patient-facing product should be able to answer these before the commit:

  • Measured p50 and p95 latency on our actual prompt length, tool chain, and peak-hour traffic, over at least two weeks
  • Ran 40+ edge-case prompts through every candidate and read the outputs for tone and usefulness, not just safety
  • Repeated the accuracy eval with demographics matched to our actual user base
  • Chose an observability stack that lets an engineer pull the full context of a specific incident within 5 minutes
  • Modeled unit cost per engaged user at projected scale, with caching and batch where applicable
  • Have a documented fallback plan for model deprecation, because every frontier vendor has killed a model with 6 months' notice

The benchmark score is the first screen, not the decision. Anyone running the review this way will probably pick a different model than the deck-ready comparison would have surfaced. That is the point.