BullshitBench: A Benchmark That Punishes Models for Agreeing With Nonsense

What BullshitBench Measures and Why It Exists

Most benchmarks reward models for producing plausible answers. BullshitBench inverts that. It measures whether a model detects that a prompt is nonsense, says so clearly, and refuses to continue with invalid assumptions. The problem it solves is the failure mode where a model confidently elaborates on a premise that has no basis in reality, such as a nonexistent framework or a misapplied mechanism. The intended audience is anyone who evaluates AI models for production use where accepting a false premise is worse than admitting ignorance. The README frames it as a measurement of clear pushback, partial challenge, and accepted nonsense. The core judgement is simple: a model that politely engages with a broken premise is not doing its job.

The v2 Question Set and Domain Coverage

The v2 set contains 100 nonsense prompts. The distribution is not uniform. Software accounts for 40 questions, while finance, legal, medical, and physics each get 15. That 40/15 split is a deliberate choice, but it means the benchmark is most sensitive to how models handle technical nonsense, and less so for other fields. The questions are generated from a draft file, `drafts/new-questions.md`, via a script called `scripts/build_questions_v2_from_draft.py`. The README lists 13 nonsense techniques, including `plausible_nonexistent_framework`, `misapplied_mechanism`, `nested_nonsense`, and `specificity_trap`. These techniques are the actual mechanisms that make the prompts hard. A model that has seen similar phrasing in training data might fail to spot the contradiction, which is exactly what the benchmark wants to catch.

How Scoring Works: Three Judges and Mean Aggregation

Every response row is evaluated by a fixed panel of three judge models: `anthropic/claude-sonnet-4.6`, `openai/gpt-5.2`, and `google/gemini-3.1-pro-preview`. The aggregation mode is `mean`, and the panel mode is `full`. The README says the legacy disagreement-tiebreak mode is retired from the main pipeline. That means every row gets three separate judgements, and the final score is the average. This design reduces the risk of a single judge being biased, but it also triples the API cost for grading. The README reports 7,440 canonical judge evaluations for the combined v1 and v2 response rows, which works out to roughly 3 judges per row. The scoring categories are clear pushback, partial challenge, and accepted nonsense. The benchmark's headline metric is the clear-pushback rate, which is the percentage of responses where the model rejects the premise outright.

Running the Benchmark: Commands and Configuration

The quick start is straightforward. You set API keys as environment variables: `OPENROUTER_API_KEY` for default routing, and `OPENAI_API_KEY` only if you route models to OpenAI directly. There are optional `OPENAI_PROJECT` and `OPENAI_ORGANIZATION` variables. Provider routing is per model, configured in `collect.model_providers` and `grade.model_providers` in the config files. The default is OpenRouter, with an example mapping like `{"*":"openrouter","gpt-5.3":"openai"}`. The main entry point is a shell script: `./scripts/run_end_to_end.sh`. Running it without arguments collects responses and runs the primary judge, which is Claude by default. To run the v2 pipeline and publish to `data/v2/latest`, you pass `--config config.v2.json` and `--with-additional-judges`. The `--serve --port 8877` flag starts a local viewer on port 8877. The viewer is an HTML file, `viewer/index.v2.html`, and you can switch between published datasets via a dropdown.

The Viewer: Charts That Reveal Reasoning Trade-offs

The v2 viewer includes six visualizations. The main chart is a leaderboard showing each model's green, amber, and red split for clear pushback, partial challenge, and accepted nonsense. There is a domain landscape view that compares overall detection against each domain. There is a detection rate over time chart that plots models by release date. Two scatter plots are particularly interesting. One asks whether newer models perform better, plotting release date against green rate. The other asks whether thinking harder helps, plotting reasoning tokens or cost against green rate. The README gives a striking example: Qwen 3.8 Max reaches rank 3 at minimal reasoning with a 94% clear-pushback rate, but drops to 26% at maximum reasoning. That is a counterintuitive result. It suggests that more reasoning can lead a model to rationalize nonsense rather than reject it. The viewer makes this trade-off visible, which is a strength of the project.

Limitations and Failure Modes

The most obvious limitation is the domain skew. With 40 of 100 questions in software, the benchmark is heavily weighted toward technical nonsense. A model that is excellent at rejecting physics nonsense but poor at software nonsense will score poorly overall. The README does not provide per-domain leaderboards, only the domain landscape view, which shows the mix but not a ranked breakdown. Another limitation is the reliance on external judge models. The scoring depends on three specific model versions, and if those models change or are deprecated, the benchmark's comparability over time is at risk. The README notes that the judge panel is fixed, but it does not say how often it is updated. The changelog entry for 2026-08-25 mentions zero new errors or refusals, which is good, but it also shows that the benchmark is actively adding models and questions, so the leaderboard is a moving target. Finally, the benchmark only measures one behaviour: how a model handles a known-nonsense prompt. It does not test whether a model can detect subtle factual errors in otherwise plausible statements, which is a different and arguably harder problem.

Alternatives and How They Differ

A common alternative is a hallucination benchmark like HaluEval or TruthfulQA, which tests whether a model produces false statements. The difference is fundamental. TruthfulQA gives models prompts that are designed to elicit false answers, but the prompts themselves are coherent and plausible. BullshitBench gives prompts that are internally incoherent or based on nonexistent concepts. The model's task is to spot the incoherence and refuse to engage. Another alternative is a sycophancy benchmark, which tests whether a model agrees with a user's incorrect premise to be agreeable. BullshitBench is close to that, but it focuses on the logical invalidity of the prompt rather than social pressure. The key difference is that BullshitBench's prompts are nonsense by construction, so there is no correct answer to give. That makes it a cleaner test of epistemic honesty.

Maintenance, Upgrade Cost, and License

The project is under the MIT license, which means you can use, modify, and redistribute it freely, with attribution. The README does not include a changelog for maintenance history beyond the v2 update, but the structure suggests a regular publish cycle. The `CHANGELOG.md` file tracks release notes and publish highlights. Upgrading from v1 to v2 requires moving to the new config and question set, but the script handles that with a flag. The main ongoing cost is API usage. Each run requires collecting responses from every model in the config, plus three judge evaluations per response. If you run the full pipeline with `--with-additional-judges`, you pay for three judge calls per row. For a large model list, that can be expensive. The README does not provide cost estimates, so you should budget based on your model count. The viewer is static HTML, so publishing is just a matter of writing files to the output directory.

Editorial conclusion

BullshitBench is for anyone who evaluates AI models for reliability in contexts where hallucination or sycophancy is costly, such as technical support, legal research, or medical triage. It is not for teams that need a general-purpose benchmark of factual accuracy or reasoning ability. Before adopting it, verify that the 100 nonsense questions match your domain coverage; the current set skews heavily toward software (40 of 100). Also confirm that your target models are available through OpenRouter or the configured provider, since collection and grading depend on external API access. The benchmark's value is its focus on clear pushback, not partial engagement, so use the green rate as your primary metric, but treat the judge panel as a cost center: every response row is evaluated by three separate models, which can add up quickly.

Official sources

1. Official documentation
2. Official README
3. Project repository