# autoreason: self-refinement that includes doing nothing

> Autoreason is a research repository from Nous Research on iterative self-refinement, built on a tournament between an unchanged incumbent, an adversarial revision and a synthesis, judged by blind Borda count. The headline claim is that a model can be taught to stop, and the repository ships the paper, the experiment runners and the blinded human evaluation files rather than a library you install.

**NousResearch/autoreason** — Autoresearch for subjective domains.

- Repository: https://github.com/NousResearch/autoreason
- Stars: 608 · Forks: 52
- Language: TeX
- License: not declared
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/nousresearch-autoreason

## The three failures of naive critique-and-revise

The paper's framing starts with a diagnosis rather than a method. Iterative self-refinement fails for three structural reasons, and the README names them: prompt bias, where models hallucinate flaws when asked to critique; scope creep, where outputs expand unchecked on each pass; and lack of restraint, where models never say no changes needed.

That third failure is the one the method is built around. A refinement loop where the model is contractually obliged to produce a change will always produce a change, even when the incumbent text was already correct. The result is the pattern the results table reports under the label that refinement destroys weak models: critique-and-revise reduced Haiku 3.5 outputs by 59 to 70 percent in word count over 15 passes.

The repository description calls the work autoresearch for subjective domains, and the title of the paper is Autoreason: Self-Refinement That Knows When to Stop. The authors are credited as SHL0MS and Hermes Agent, and the paper itself is included in the repository as a compiled PDF alongside the LaTeX source.

What follows is an unusually complete research artifact. There is a human_eval/ directory with blinded evaluation files, a tasks/ directory with the prompts, an experiments/ tree with the runners and the raw results per configuration, and a paper/ directory. The last push to the repository was on 2026-04-12 and there are no GitHub releases, so this is a snapshot of a study rather than a versioned tool.

## The tournament: incumbent, revision, synthesis, and do nothing

The method is a tournament with a built-in abstention. Each iteration produces three competing versions: the unchanged incumbent, labelled A; an adversarial revision, labelled B; and a synthesis of the two, labelled AB. Those three are judged by fresh agents that share no context, and the winner is decided by blind Borda count. Do nothing is always a first-class option.

The flow, as the paper diagram has it:

```
Task Prompt → Incumbent A
                  ↓
        ┌─── Critic (fresh agent) ───→ Critique
        │
        ├─── Author B (fresh agent) ──→ Revision (B)
        │
        └─── Synthesizer (fresh) ─────→ Synthesis (AB)
                  ↓
          Judge Panel (3 fresh agents, Borda count)
                  ↓
              Winner → new A  (or converge if A wins k=2 times)
```

Two details in that diagram carry most of the design. Every participant is a fresh agent, so the critic does not know what the incumbent looked like before and the judge panel does not know which candidate is which. Freshness is the mechanism that makes the judgement blind, and the results table reports a separate finding for judge count: seven judges give 3 times faster convergence than three judges, and one judge is noisy and slow.

The convergence rule is the second detail. If the incumbent A wins twice, the loop stops. That is the entire answer to the lack of restraint problem, and it is a change to the stopping condition rather than to the prompt. A method that revises until the incumbent wins twice will terminate on a text that is already good, and a method that revises until it looks better than the incumbent never will.

The paper contents list shows the scope of the experiments behind this: 5 open-ended tasks, 3 constrained tasks, 4 baselines and 15-pass iterations for the writing experiments, and 150 CodeContests problems across 3 strategies and 4 model tiers for competitive programming.

## What the results table actually claims, and where the caveats are

The key results table is dense and worth reading closely, because some rows are wins and some are warnings.

The strongest claim is the 42/42 perfect sweep, where Haiku 3.5 with autoreason scored perfect Borda across 3 tasks and all baselines degraded below single-pass. The competitive programming result is 77% against 73%, with Sonnet 4.6 on 150 CodeContests problems measured on private tests, autoreason against single-pass. A compute-matched comparison gives 40% against 31%, autoreason against best-of-6 sampling on Haiku 3.5 over 150 problems, which is the row that matters most for a reader worried that the gains are just extra sampling.

Then the caveats. There is a transition point at Haiku 4.5, where at 60% private accuracy the held-out gains vanish because the generation-evaluation gap has closed. That is a specific and important claim: autoreason helps most when the model is bad enough that its own evaluation is still informative, and stops helping once the model is strong enough to judge itself well. The scaling curve in the same row reads Haiku 3.5 at 40%, Haiku 4.5 at 60%, Sonnet 4 at 64% and Sonnet 4.6 at 77% on private tests with autoreason.

The length-controlled row addresses the obvious objection to any self-refinement method, that longer output wins. Autoreason beats 3 of 4 baselines even at matched word count, with 21 wins out of 28.

And the component row is the most interesting for anyone thinking of building this: removing either B or AB collapses the tournament, with convergence in 2 to 3 passes instead of 24. Both the adversarial revision and the synthesis are load bearing. A system that only criticises and revises, which is the common design, is not a simplified version of this one, it is a different and much worse system.

## The experiment tree, and what you would actually rerun

The repository structure section reads as an inventory of what the authors ran, and the v2/ directory under experiments/ is where the work lives.

The runners are separate scripts per experiment family. run_overnight.py is the main experiment runner for the writing tasks, run_code_overnight.py handles the CodeContests problems, run_code_haiku45.py is a separate runner for the Haiku 4.5 code experiments, run_multi_seed.py handles the multi-seed replication, and run_ablations.py covers the component, judge, aggregation and length ablations. compute_stats.py does the statistics, described as bootstrap confidence intervals and McNemar tests.

The results directories are then organised by experiment, which makes it possible to look at one question at a time. There are separate directories for the Sonnet 4.6 code results over 150 problems, the Haiku 3.5 code results over 150 problems, the Haiku 4.5 code results over 150 problems, and a best-of-N compute-matched control. There are directories for the 15 independent writing runs, the ablations, the baseline comparison outputs, the multi-task runs, the Monte Carlo replication over 5 runs, and the constrained task and scaling remedy experiments.

That best-of-N directory is the one to look at first if you want to check the central claim, because it is the compute-matched control against the 40% versus 31% row.

The paper contents list gives the rest of the experimental design. There is a 5-tier model scaling comparison running from Llama 8B through Gemini Flash, Haiku 3.5, Haiku 4.5 and Sonnet 4. The ablations cover judge count of 1, 3 and 7, Borda against majority aggregation, component necessity and length-controlled evaluation. The replication work includes Monte Carlo over 5 runs and multi-seed replication of 15 runs across 5 tasks. The failure analysis section lists 8 remedy experiments for the Sonnet 4.6 scaling failure, plus a failure taxonomy.

The primary language of the repository is recorded as TeX, which is an accurate signal that this is a paper repository with the code as supporting evidence, not a code project.

## The human evaluation and the citation

The human evaluation is the part of the design that makes the Borda numbers easier to believe, and it is a separate directory with a separate answer key.

Blinded files for human raters live in human_eval/, covering 5 tasks times 3 methods: autoreason, critique-and-revise and single-pass. The method labels are replaced with randomized 4-character codes, so the rater does not know which pipeline produced the text, and the rubric and instructions are in human_eval/README.md. The answer key that maps codes back to methods is a separate file, human_eval_answer_key.json, at the top level of the repository, which is the usual arrangement for keeping the mapping out of the rater's way.

Including critique-and-revise and single-pass as comparison methods matters, because those two are exactly the baselines that a self-refinement paper needs to beat. Single-pass is the do-nothing control, and critique-and-revise is the naive loop that the whole design is an argument against.

The citation block is a plain BibTeX entry for a journal article:

```
@article{shl0ms2026autoreason,
  title={Autoreason: Self-Refinement That Knows When to Stop},
  author={SHL0MS and Hermes Agent},
  year={2026},
  url={https://github.com/NousResearch/autoreason}
}
```

Two things are worth noting about the state of the repository. It has no GitHub releases, so there is no version to cite or install, and the licence is not stated in the metadata, which matters if you want to reuse the code rather than read it. The paper PDF and the LaTeX source are both in the repository, so the write-up itself is available offline.

The tasks/ directory holds the task prompts, 5 open-ended and 3 constrained, which is the piece you need if you want to run the method on your own writing problems rather than the paper's.

## Conclusion

Treat autoreason as a paper and experiment harness to read and reproduce, not as a library to depend on, since the repository ships LaTeX sources, task prompts, experiment runners and evaluation files rather than a package. The finding worth taking away is the negative one: refinement hurts weak models, with critique-and-revise shrinking Haiku 3.5 outputs by 59 to 70 percent in word count over 15 passes. Before drawing your own conclusions, read the ablation results in results_ablations/, where removing either the revision or the synthesis collapses the tournament, and check the compute-matched control in results_code_best_of_n/ so you are not reading a win that is just extra sampling.

## FAQ

### What is the core idea behind autoreason?

Each iteration produces three competing versions, the unchanged incumbent A, an adversarial revision B and a synthesis AB, judged by fresh agents with no shared context via blind Borda count. Doing nothing is always a first-class option, and the loop converges when the incumbent wins twice.

### Why does naive critique-and-revise fail?

The README names three structural reasons: prompt bias, where models hallucinate flaws when asked to critique, scope creep, where outputs expand unchecked on each pass, and lack of restraint, where models never say no changes needed. Critique-and-revise reduced Haiku 3.5 outputs by 59 to 70 percent in word count over 15 passes.

### Does autoreason beat plain sampling at the same compute?

On Haiku 3.5 over 150 problems, autoreason scored 40% against 31% for best-of-6 sampling at matched compute, and the results are kept in the results_code_best_of_n directory as the compute-matched control.

### When does autoreason stop helping?

At the Haiku 4.5 transition point, where private accuracy reaches 60% and the held-out gains vanish because the generation-evaluation gap has closed. The reported code scaling curve runs Haiku 3.5 at 40%, Haiku 4.5 at 60%, Sonnet 4 at 64% and Sonnet 4.6 at 77%.

### Is autoreason a library I can install?

No. It is a research repository containing the paper PDF and LaTeX source, task prompts, the experiment runners under experiments/v2/, the per-experiment results directories and the blinded human evaluation files, with no GitHub releases.

## Sources

- [Issues](https://github.com/NousResearch/autoreason/issues)
- [NousResearch/autoreason on GitHub](https://github.com/NousResearch/autoreason)
- [README](https://github.com/NousResearch/autoreason/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/nousresearch-autoreason