Autoresearch for Claude Code: Karpathy's overnight improvement loop, generalised to any metric

Where the idea comes from: Karpathy's autoresearch

The README starts from Andrej Karpathy's autoresearch, which it describes as a 630-line Python script that improved machine learning models on its own overnight, running 100 experiments per night. What made that work, in the README's summary, were a few simple principles: one metric, a constrained scope, fast verification, automatic rollback and git as memory.

This project's claim is that those principles are not specific to machine learning. It generalises them to any domain where there is a number to measure, and the README names code, content, marketing, sales, HR and DevOps. Its own tagline puts it plainly: set the goal, the agent runs the loop, you wake up to results. The target users are people who already let Claude Code, OpenCode or Codex work unattended and want that time spent on measured improvement rather than open-ended edits.

How Autoresearch works: the loop and its eight rules

Before it loops, the agent does a one-time setup: it reads all in-scope files, extracts or asks for a mechanical metric, decides which files may be modified and which are read-only, runs the verification once to set a baseline as iteration zero, and shows you the setup before starting. The loop itself is short, and the README gives it in full:

LOOP (N iterations or until done):
  1. Review current state + git history + results log
  2. Pick the next change (based on what worked, what failed, what's untried)
  3. Make ONE focused change
  4. Git commit (before verification)
  5. Run mechanical verification (tests, benchmarks, scores)
  6. If improved → keep. If worse → git revert. If crashed → fix or skip.
  7. Log the result
  8. Repeat until N iterations complete or goal is met.

Progress is logged in TSV format. Eight rules govern the loop. Runs are bounded by default, and unlimited iterations are opt-in. The agent reads before it writes, makes one change per iteration, and accepts only mechanical verification, never a subjective judgement that something looks good. Failed changes revert automatically, equal results with less code win, experiments are committed with an experiment: prefix so git log and git diff act as memory, and when stuck the agent is told to re-read, combine near misses or try a radical change.

Using Autoresearch with Claude Code: goals, metrics and commands

Version 2.2.0 added an autonomous orchestrator. You type a plain-language goal to /autoresearch, and it classifies the goal, derives a success predicate, confirms it with you once, and then loops across subcommands until the goal is met, without you chaining commands by hand. If you prefer the original behaviour, invoking it with Metric: and Verify: still runs the classic loop unchanged. The README points to guide/autoresearch-orchestrator.md for the details.

The subcommands cover a whole delivery path. /autoresearch:plan sets the goal, metric and scope; /autoresearch:debug and /autoresearch:fix find and repair errors; /autoresearch:security works through STRIDE, OWASP and red-team checks; /autoresearch:ship stages, deploys and releases. Others handle requirements probing, edge-case scenarios, a five-expert prediction swarm, structured debate, documentation, research write-ups, evaluation analysis and regression checks against a baseline. The repository ships packaging for each host, in .claude-plugin/, .opencode/ and .agents/, alongside the guide/ and docs/ directories.

Hooks and safety: what is protected, and where

The README is careful about what its guardrails are. Hooks are described as defense in depth, not a security sandbox, and they exist only in Claude Code; OpenCode and Codex get the core skill, runtime and installation surface without hook parity.

The hooks listed run before tool use. scout-block keeps directories such as node_modules/, .git/ and __pycache__/ from filling the agent's context. privacy-block stops the session from reading .env files, SSH keys and credentials. dangerous-cmd-block blocks force pushes, rm -rf and similar destructive commands.

That split matters for anyone choosing a host. A loop that commits and reverts on its own for hours is exactly the situation where a blocked destructive command is worth having, and on OpenCode or Codex you would need to supply equivalent protection yourself, for example by running the loop in a disposable checkout or container.

Token cost and the v2.1 rebuild

A loop that runs a skill hundreds of times is sensitive to how large that skill is. The README says the v2.1.0 release was a major architecture rebuild for that reason. The earlier single SKILL.md was 813 lines and cost about 100,000 tokens per invocation. It was replaced by a 41-line routing file and twelve self-contained command files of 94 to 120 lines each, costing roughly 5,000 to 8,000 tokens per invocation. The README calls that a 95% token reduction with the same capability surface.

For anyone on a usage-limited plan, this is the most practical detail in the README. An overnight run is only useful if it does not exhaust the session window in the first hour, and a routing file that loads only the command it needs is what makes long runs affordable.

Releases, maintenance and when not to use it

The recent releases are v2.2.0, the autonomous orchestrator, on 2026-06-20; v2.2.1, orchestrator seam hardening, on 2026-06-23; and v2.2.2, a stabilisation release, on 2026-08-12, which is also the date of the last push. The project is MIT licensed, and the repository includes tests/, a CONTRIBUTING.md and a COMPARISON.md.

Autoresearch fits problems with a fast, trustworthy number. It fits badly where the only honest judge is a person, such as design quality or the tone of a piece of writing, because rule four forbids exactly that kind of verification, and a proxy metric the agent can game will be gamed over a hundred iterations. It also assumes git history you are happy to fill with experiment commits, so run it on a branch and squash or discard the history once the result is in.

Editorial conclusion

Use Autoresearch when you have a goal you can measure with a command, such as a test pass rate, a benchmark or a score, and a codebase in git that can absorb many small experimental commits. Start with a bounded run on a narrow scope, read the results log before trusting the outcome, and keep the Claude Code hooks on, since they are the only guardrails the other hosts do not get.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. uditgoenka/autoresearch on GitHub