# keystone-bench: does a clinical assistant's advice move when its evidence does

> Keystone takes HealthBench conversations, pulls out the single fact a clinical answer rests on, and rewrites it, then checks whether the assistant's recommended action follows the change. It ships as a dataset, a runner and an audit trail rather than a score, and it is explicitly built to test rubric graders as well as models.

**xinxuxin/keystone-bench** — A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.

- Repository: https://github.com/xinxuxin/keystone-bench
- Website: https://xinxuxin.github.io/keystone-bench/
- Stars: 931 · Forks: 3
- Language: Python
- License: NOASSERTION
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/xinxuxin-keystone-bench

## The gap it names is in the rubric, not in the reply

A clinical chat benchmark scores a reply against a physician-written rubric for the message the physician saw, and the README is clear that this says nothing about what the assistant does when a decisive fact is missing, replaced or contradicted, and nothing about whether the rubric still measures anything once it is. That is the problem Keystone is built to make measurable. The framing in one line is that it pulls the one fact an answer rests on and sees whether the action moves with the evidence. The consequence is that the benchmark is about behaviour under perturbation rather than about knowledge. A model can score well on a HealthBench rubric by producing a well-formed answer to the question as posed. Keystone asks a different question, which is whether the answer would have been different if the fact had been different. For anyone building a clinical decision support product, that is the question that matters, and it is not one the existing rubric structure can express.

## Every item is a twin pair that changes exactly one thing

The unit of evaluation is a pair. Every HealthBench source conversation carries a decision frame, which is what is being decided, the action the stated evidence supports, and the facts it rests on, and it is paired with twins that change exactly one thing, each annotated with the evidence state after the edit along with the actions a clinician would accept or forbid on it. There is also a paraphrase-only control that changes nothing. The README shows the shape with an illustrative vignette, and the same example appears in the released vignette as a triple: an original that mentions not being pregnant, a twin with one sentence gone so that two actions become possible, and a control that is reworded with nothing removed. The expected behaviour on the twin is to ask about pregnancy or to answer conditionally on it, and the measured behaviour is whether the reply stayed definitive on the twin without naming the gap, and whether the control left it unchanged. That pairing is the design. The twin catches overconfidence and the control catches a model that merely changed its tone on a rephrasing.

## Nine families, two of which are negative controls

The families are the taxonomy of edits, and reading them tells you what the benchmark considers important. Four apply to every source: missing_evidence removes one decisive fact, conflicting_evidence adds one credible contradiction, demographic_shift changes one attribute that should change the advice, and salient_distractor adds one salient but irrelevant fact, where correct behaviour is no change. That last one is a negative control, and demographic_control is a second one, adding an attribute a clinician would not act on, chosen to tempt an adjustment. Five more apply only where the source allows: alternative_evidence puts the same fact back with a value supporting a different action so one decision is seen in three evidence states, buried_red_flag mentions a red flag in passing so the correct answer becomes urgent evaluation, missing_evidence_early removes the load-bearing fact from an earlier turn far from the question, and from version 0.7.0 alternative_evidence_hard moves one value the decision rests on just across a named clinical threshold so the correct action changes with nothing in the message pointing at it. Two negative controls among nine families is a deliberate proportion. The benchmark is not only interested in catching failure, it is interested in catching a model that overreacts.

## Layers, splits and a fixed quick set

The same items appear in several layers, and this is the part to read carefully before quoting a number. There is all, then core, which keeps items where three raters' median materiality is what the family requires and there is no mechanical defect, then strict, which adds the requirement that the edit is tied to a rubric criterion and that not both reviewer models rejected it, then primary, which adds that the annotated evidence state agrees with what the family's edit is designed to do and that the second reviewer did not disagree. The action outcomes use the primary layer. There is also quick, a fixed set of 350 twins made up of 40 strict twins with a control per family, 39 on missing_evidence_early and 31 on alternative_evidence_hard, and the same twins are carried from one release to the next so results stay comparable. Sources are split 80/20 into dev and test by a hash of the source id, so the twins and controls of one source never straddle the split. The design implication is that a paper reporting a Keystone number has to say which layer it used, and two numbers on different layers are not comparable.

## Outcomes are actions, graded by a judge from another vendor

A judge from another vendor maps each reply to the annotation, so the primary outcomes are about actions rather than about similarity to a reference answer, and every judged item is in the denominator rather than only the ones that could be graded. Alongside that, the physician rubric's applicability is scored alongside the model, which is the second half of the project: whether the rubric still applies once the evidence has moved. The README gives a grader validity figure for that judge. Six authored replies with intended labels per item, 166 items in total, and the action judge separates unsupported commitment from correct replies at 0.90 to 0.99 with GPT-4.1 as judge, with the details in docs/JUDGE_CHECK.md. The annotation itself is described as model-drafted and independently reviewed by another vendor, marked tier: silver, with clinician-confirmed rows becoming gold. So the labels are not all physician-written, and the tier markers exist precisely so a user can tell which rows carry that weaker provenance.

## What is in the release, and what you can check

The release numbers give the scale. There are 7,419 twins over more than 1,260 HealthBench conversations, of which 1,234 come from seven physician-agreed strata with four twins per source plus the additional families, and 26 more carry a threshold twin. 2,861 are multi-turn, with every untouched turn byte-identical to the source. The paraphrase control covers 1,132 sources, reworded with nothing added or removed and released only after a model fidelity check passes. Each twin records what changed, why it is load-bearing, the expected safe behaviour, three materiality ratings with rationales, two naturalness verdicts, which rubric criteria depend on the edit, and mechanical quality flags. There is a reference set of 153 conversations that naturally lack decisive information, unperturbed, and reference results for five assistants with every reply and its classification shipped in release/reference_records.jsonl. Auditability is a stated feature, with a published SHA-256 per file that your own rebuild is checked against, and the README sentence listing the tracked files is cut off in the copy available here, so the exact count of files with published hashes is not stated in what can be read. The docs folder is where the detail lives, with a data card, a schema, a protocol, a hypothesis lineage, results, a judge check, a shortcut audit, a policy matrix, a cross-scoring write-up, a rubric consequence document and a frame versus physician comparison.

## Running it means two models, a rebuild step and a fixed set of checks

The Makefile is the entry point, and the targets say a lot about how the project wants to be used. The data target fetches HealthBench, replays the shipped edits and rebuilds the distribution byte for byte, and it runs the build twice, once to build and once with --check:

```bash
data:            ## fetch HealthBench, replay the shipped edits, rebuild dist/ byte for byte
	$(PY) tools/build_release.py && $(PY) tools/build_release.py --check
```

The check target runs the mechanical quality checks, the degenerate-strategy test and the published hashes, ending in keystone build --check and keystone validate. The test target is the test suite and needs no keys, and the quick target is described as the first run, the fixed quick set across every family:

```bash
quick:           ## first run: the fixed quick set, every family
	keystone run --family all --layer quick --model $(MODEL) --judge $(JUDGE) --out runs/quick__$(subst /,_,$(MODEL))
```

The Makefile defaults the model to openrouter/openai/gpt-5.6-terra and the judge to openrouter/anthropic/claude-sonnet-5, which tells you the design assumes a separate judge model from a different vendor. The anchors target is worth noting, since it covers validity against behaviour, the physicians' rubric, their ideal answers and the edit fingerprint with no model calls at all. The package itself is on PyPI as keystone-bench with a keystone console script, one runtime dependency on httpx, and optional extras for inspect-ai and for a dev set of pytest, numpy, scipy and statsmodels. It requires Python 3.10 or newer and is marked Development Status 4 - Beta.

## A young release line with a custom licence to read

The release history is short and recent, with v0.7.0 on 2026-10-19 and v0.4.0 on 2026-09-07, and the repository is not archived with the last push on 2026-10-19. The version series is in the 0.x range, so treat minor bumps as potentially breaking for a dataset, since an item set or a labelling scheme can change between versions, which is exactly why the quick set is fixed across releases. On licensing, the licence field on the repository is recorded as a custom one that GitHub cannot classify, while the pyproject.toml declares license MIT and the README badge reads MIT, and there is a LICENSE file at the repository root along with a THIRD_PARTY-style NOTICE in the tree. Because the metadata classification and the manifest disagree, the file itself is what to read before publishing anything derived from the dataset. The repository also carries a .zenodo.json for archival, a CITATION.cff, a SECURITY.md, a CODE_OF_CONDUCT.md and a CONTRIBUTING.md, and the project page is published in English, Chinese, Spanish and Japanese, which tells you the intended audience includes clinical researchers outside an English-only setting.

## Conclusion

Adopt keystone-bench if you are evaluating a clinical decision support assistant and you want to know whether its advice degrades when the evidence underneath it changes, and if you can run two models, one to answer and one to judge. Do not adopt it as a replacement for a clinician review, since its annotations are model-drafted and independently reviewed rather than clinician-confirmed except on some rows, and do not treat it as a general-purpose medical benchmark, because it is built on HealthBench conversations and inherits their scope. Before you commit, read docs/DATA_CARD.md, docs/SCHEMA.md and docs/PROTOCOL.md, and decide which layer you will report, since the primary, strict, core and quick layers give different numbers for the same items.

## FAQ

### What is keystone-bench?

It is a clinical decision-evidence benchmark for chat assistants, built on HealthBench source conversations. Each source carries a decision frame and is paired with twins that change exactly one thing, plus a paraphrase-only control, so a benchmark can measure whether the assistant's action moves with the evidence.

### How do I run the keystone-bench quick set?

The Makefile has a quick target described as the first run, the fixed quick set of 350 twins across every family. It calls keystone run with --family all --layer quick and takes a --model and a --judge, with the Makefile defaulting to openrouter/openai/gpt-5.6-terra and openrouter/anthropic/claude-sonnet-5.

### What is the difference between the Keystone layers?

core keeps items where three raters' median materiality matches the family and there is no mechanical defect, strict adds that the edit is tied to a rubric criterion and not both reviewer models rejected it, and primary adds that the annotated evidence state agrees with the edit's intent and the second reviewer did not disagree. The action outcomes use the primary layer.

### Are the Keystone annotations written by physicians?

The decision-evidence layer is described as model-drafted and independently reviewed by another vendor, marked tier: silver, with clinician-confirmed rows becoming gold. The source conversations come from HealthBench, and the judge validity check is documented in docs/JUDGE_CHECK.md.

## Sources

- [Issues](https://github.com/xinxuxin/keystone-bench/issues)
- [Project website](https://xinxuxin.github.io/keystone-bench/)
- [README](https://github.com/xinxuxin/keystone-bench/blob/main/README.md)
- [Releases](https://github.com/xinxuxin/keystone-bench/releases)
- [xinxuxin/keystone-bench on GitHub](https://github.com/xinxuxin/keystone-bench)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/xinxuxin-keystone-bench