The 8B SLM That Beats Frontier AI on HealthBench

Executive Summary

We trained an 8B parameter Specialised Language Model (SLM), and evaluated its performance on the hardest physician-validated context-seeking subset of HealthBench: a benchmark designed by OpenAI to measure models' performance on real-world healthcare scenarios.

• Score: Our SLM scores 96.5%. Opus 4.6 scores 18.1%, and Sonnet 4.6 scores 26.3%.
• Stress test: Each request is run 16 times, and the weakest result is reported as a measurement for worst-case performance. Our SLM retained a 72.4% score. Every other LLM fell below 10%.
• Cost: At 1 million queries per month, our SLM would save US$146,000 to US$407,000 annually.

The results reveal that despite their small size, SLMs trained with the right methodology can deliver higher scores, better reliability, and at far lower costs than general-purpose frontier LLMs.

Methodology

What is HealthBench?

HealthBench is a benchmark developed by OpenAI to measure frontier models' performance in healthcare settings. It was designed to address the limitations of prior healthcare benchmarks, which were not representative of real-world use cases and often relied on multiple-choice questions.

OpenAI worked with 262 physicians from 60 countries to develop 5,000 realistic multi-turn conversations, then scored them against 48,562 physician-written criteria. Two properties make it the right benchmark to test against. First, latest frontier models perform relatively poorly, so there is real room to improve. Second, it tests open-ended reasoning, the kind of skill that matters in real-world situations.

Experiment Subset

We deliberately chose the hardest, most rigorously validated subset of HealthBench for our experiment, concretely:

Multiple physician validated: Multiple physicians independently agreed on the grading criteria. This removes annotation noise and ensures that our scores reflect the judgement of real clinicians, not an LLM.

The hardest 20%: We only tested against the most difficult 20% questions in HealthBench. Easy questions are already near-solved. The hard cases are where the benchmark still has enough gradient to show meaningful progress.

Open-ended questioning: Tests whether a model knows when to stop, ask for clarification, and generate a follow-up question. The HealthBench paper reveals that this is where frontier LLMs perform the worst.

Training Details

Training data: We used publicly accessible medical data to generate synthetic cases using our proprietary methods.

Compute time: Total model training time took under 28 hours on a single L40S GPU.

Compute cost: The total training cost was just shy of US$40.

Base model: Qwen3-8B, an open-weight model from the Qwen3 family.

Results

Score

Our model scores 96.5%. Anthropic's strongest result is Haiku 4.5 at 31.2%, while Opus 4.6 reaches 18.1%. The Qwen3-8B base model starts at 2.9%, so the lift comes from the training process.

Inference Cost

At 1 million queries per month, the inference bill for using our model is around US$300. Sonnet 4.6 would cost about US$12,500, with Opus 4.6 near US$20,900. [2]

That is about US$146,000 to US$247,000 in annual savings, with higher scores and the operational freedom to deploy, tune, and monitor the model inside the organisation’s own environment.

Reliability

In a safety-critical domain like health, the consequences of one bad model response can outweigh the benefits of many good ones. Average scores only tell you what happens on one random draw. The stress test asks a different and more salient question: if the same model processes the same problem many times, how bad is the worst answer likely to be?

Our trained model stays dependable under real workload pressure. It starts at 96.5% and drops to 72.4% after 16 rounds of testing.

Every other model's performance collapses catastrophically. Haiku 4.5 drops from 31.2% to 2.6%, Sonnet 4.6 drops from 26.3% to 5.3%. A model that demos well but has a fast-collapsing reliability curve is fragile in production, especially in agentic systems with retries, branches, or multiple tool calls.

Why Do Frontier Models Struggle Here?

Our analysis points to three main reasons.

First, frontier LLMs' training incentives are misaligned with this context-seeking behaviour.

The benchmark rewards restraint: identify what is unknown, ask for the missing clinical context, then give advice.

Frontier LLMs are trained to be helpful, which biases them toward giving a response immediately, even when key clinical information required to make the correct judgement is never surfaced. [1]

Our model was trained to behave like an expert: recognise gaps, ask the right follow-up questions, and only provide advice after clarifying the scope of the problem.

Second, the very capability that makes frontier models impressive also makes them too eager to answer.

We were surprised that Opus 4.6 scores below Haiku 4.5, despite being the more capable model. Our trace analysis suggests that Opus is more confident, more comprehensive, and less willing to stop at a clarifying question.

Prompt engineering was not enough to steer the model to behave differently. But training was. The experiment shows that, with the right methodology, we can move a model away from its default response pattern and teach it to adopt a new behaviour that is more appropriate for the real-world environment it operates in.

Third, Opus 4.6 did not answer 10.8% of valid clinical questions because its guardrails treated the questions as policy violations.

This automated blocking behaviour highlights the operational constraint of general LLMs. Their guardrails come pre-set by the provider, so valid domain-specific questions can be blocked under a generic policy. An SLM gives the organisation freedom to deploy guardrails that match its own risk profile, legal requirements, and operational environments.

Conclusion

Frontier LLMs are trained to answer. What they do not do is stop and ask whether they have enough information first, and on the cases where physicians agreed asking was the right move, that gap is the failure.

We set out to determine if we could train that behavoiur into a model, so it matches how physicians rate the ability to seek context before answering. We took a small model that performed at 2.9% and trained it to score 96.5%, scoring 72.4% on the worst run of 16.

This improvement came from applying training stages that mirror how people learn a skill, not from prompting. The training steps, the prompts and reward function we used, are shown in full in the technical report.

If you are evaluating whether a specialised model fits a workflow of your own, we would be glad to look at it with you.

[1]: This "eagerness to respond" is so ingrained that we found prompt engineering was insufficient to steer the LLMs away from this behaviour.

[2]: The updated comparison also includes GPT-5.5 xhigh: it lands close to Opus 4.6 on score, but costs about US$40,253 at 1 million queries per month.