autoresearch: an agent rotation that implements an issue until it scores well enough to merge

The idea comes from karpathy, applied to any repository

The project is built on the ideas of karpathy/autoresearch, as the README states, and its aim is a general purpose fully automatic development tool. The name of the repository description is autodevelop: autoresearch for software development, and the README is explicit about the promise in its opening line, that a fully automatic development tool means you only have to worry about tea and sleep, and that by morning the features are implemented automatically and at high quality.

The scope claim is that it works for any Git project, with Go, Node.js, Python, Rust and Java named as examples, and that it supports several issue sources.

The most convincing evidence in the repository is self-referential. The README says the project was used to implement its own Issue #2, with a recorded terminal session linked as an asciicast, and the workflow section gives a worked example of 19 issues, numbered 22 through 40, going from a product requirements document to implementation for the project's own desktop app.

That is worth pausing on, because it is also the main limitation. A tool that closes an issue by writing code, running the build and merging a pull request is a tool that acts on a repository without a human in the loop, and the repository that demonstrates it best is one whose author controls both the tool and the tests it grades against. The primary language of the repository is Shell, the licence is MIT, and there is a single tagged release, v0.1.0, described as the autoresearch desktop release, dated 2026-04-27.

The scoring gate is what stops the loop early

The architecture table in the README is a single table of components, and it makes the flow obvious.

The issue source, whether GitHub Issue, a local Issue, Baidu iCafe or Aliyun Codeup, is the trigger input. run.sh is the core runner. Claude, Codex, OpenCode, Claude-Mimo and DeepSeek are the agents that rotate for review. Then a score question, score at least 85, is a scoring gate. If the result is a pass, a pull request is created automatically, merged, and the issue closed. If it is a fail, it goes into the next round of iterative fixing.

The rotation formula is given as (iter - 1) % N, where N is the number of enabled agents, so the first agent does the initial implementation and the rest take turns reviewing and fixing.

The workflow section spells out the two phases. Iteration 1 is the initial implementation by the first agent in the list passed with -a. Iterations 2 and up have all the enabled agents review and fix in order. Once the score reaches the threshold, which defaults to 85, the tool creates the pull request, merges it, comments and closes the issue.

The quality gate itself has two parts, and the second is the more interesting one. The quality gate is Build/Lint/Test plus an LLM score of at least 85. So the score is not the only check; the project's own build, lint and test commands have to pass before the score is even considered. An LLM rating a diff is weak evidence on its own, but an LLM rating a diff whose build and tests pass is a much stronger signal, and combining the two is what keeps the loop from merging code that merely looks right.

Running an issue, and the flags that control the loop

The quick start is a shell script with a positional issue number, and the flags are what make it usable in a real workflow.

git clone [email protected]:smallnest/autoresearch.git
autoresearch/run.sh -p /path/to/project 10
autoresearch/run.sh -p /path/to/project 10 16
PASSING_SCORE=90 autoresearch/run.sh 10
AGENT_MODEL=opus autoresearch/run.sh 10
autoresearch/run.sh -a claude,codex 10
autoresearch/run.sh -c 42 10
autoresearch/run.sh --issue-source=codeup --codeup-token=xxx --codeup-org=123 --codeup-repo=456 42

The second positional argument is the maximum number of iterations, so a run with 16 gives the agents sixteen chances to reach the threshold. The first agent in the -a list does the initial implementation, and -c continues work on an existing issue with a fresh iteration budget.

Two model variables exist, and the precedence is documented. AGENT_MODEL sets the model for all agents, while a per-agent variable such as AGENT_MODEL_CLAUDE overrides it for one agent, with the per-agent setting taking priority over the global one. That is how you run a cheap model for the bulk of the work and an expensive one for the final review.

PASSING_SCORE changes the gate itself, and the default of 85 is the value the workflow section refers to.

The script takes the issue number as its argument in every mode, which is the one piece of interface that stays identical across GitHub, local files, iCafe and Codeup.

The prerequisite CLIs decide which mode you can run at all

The prerequisites are listed as tools to install as needed, and the list is a good map of the integrations.

which claude            # Claude Code CLI
which codex             # OpenAI Codex CLI
which opencode          # OpenCode CLI
which deepseek-tui      # DeepSeek CLI (npm install -g deepseek-tui)

The coding agents are external command line programs, not Python libraries the tool imports. Claude Code, the OpenAI Codex CLI, OpenCode and the DeepSeek TUI are each invoked as a subprocess, which is why the checks are which commands. The DeepSeek entry notes the npm install -g deepseek-tui command it needs.

Alongside them, each issue source has its own client. GitHub mode needs the GitHub CLI, checked with gh auth status, and the Baidu mode needs the iCafe CLI and the iCode CLI, checked with icafe-cli whoami and icode-cli whoami. The Codeup mode uses curl.

The project also needs the build tools for the language of the project it is working on, with Go, Node, Python, Rust and Java named. That requirement is not incidental: the quality gate runs the project's build, lint and test commands, so without the toolchain the gate cannot pass and the loop cannot finish.

The repository layout reflects the same split. There is run.sh for the core runner, run_all.sh alongside it, an agents/ directory for the agent definitions, a lib/ directory for the shared shell logic, a skills/ directory for the workflow skills such as the PRD generator, and codeup_cli.py as the Codeup integration written in Python rather than shell.

Local issue files, and a mode that never touches the remote

The local issue mode is the one that changes how the tool can be used, because it removes GitHub from the equation entirely.

The setup is a directory and a file.

mkdir -p .autoresearch/issues
./run.sh 8
./run.sh --issues-dir=.autoresearch/issues 8
./run.sh -p /path/to/project --issues-dir=.autoresearch/issues 8

The file naming convention is issue-NNN-description.md, with the number zero padded to three digits, and the CLI number matches both the padded and unpadded form, so a run for 8 finds issue-008-*.md or issue-8-*.md. The first heading line is the issue title and the rest is the body. The location is $PROJECT_ROOT/.autoresearch/issues/ by default, and --issues-dir overrides it.

What happens at the end of a local run is the important difference. The code is committed to a local branch, without pushing to the remote, without creating a pull request and without merging. The result is appended to the end of the issue file itself, with the score, the iteration count, the implementation method listing the agents used, the branch name, the completion time, and a log tail.

The mode table makes the three-way difference concrete. In GitHub mode the issue comes from gh issue view, the gh CLI and a GitHub remote are required, and push, pull request and merge all happen automatically. In local mode none of that is required and none of it happens. In Baidu iCafe mode the issue comes from icafe-cli card get, the icafe-cli and icode-cli tools are required, and the flow uses push_cr and submit_review rather than the GitHub equivalents.

The two modes are selected automatically. If a matching local file exists the tool enters local mode, and if there is none it takes the GitHub path, so the same project can mix both.

Baidu iCafe and Aliyun Codeup, and the same loop on other trackers

The Baidu mode exists because the GitHub pull request flow does not map onto how that organisation works internally, and the adaptation is complete rather than partial.

The issue comes from a card read with icafe-cli card get, the code is pushed for review with icode-cli git push_cr, the merge is icode-cli api submit_review, the card is closed with icafe-cli card update, and comments are posted with icafe-cli comment create. The commit message format changes too, from the feat: implement issue #N style to a space, sequence, bracketed type and title format.

There is also a scoring difference in that path, where the iCafe flow shows a score plus 2, so the tool's own scoring is not identical across modes.

Auto-detection is defined precisely. Baidu mode activates without the flag if two conditions hold, the Git remote contains icode.baidu.com, and either the --space parameter or the ICAFE_SPACE environment variable is set.

export ICAFE_SPACE=cloud-iCafe
./run.sh --issue-source=baidu --space=cloud-iCafe --target-branch=develop 22210

The Codeup mode is the fourth source and works the same way, with its own token, organisation and repository arguments passed on the command line.

The reasons to care about these modes are practical rather than exotic. If your organisation runs its issue tracker in a system where pull requests are called code reviews and issues are called cards, a tool that speaks only the GitHub vocabulary needs you to write a bridge. These modes are that bridge, and the difference in approach from the GitHub path is a mapping of the same five operations, not a different algorithm.

Where the automatic loop is the wrong tool

There are three limits, and the first is not a defect but a design decision you have to accept or reject.

The default path merges the pull request and closes the issue without asking a human. The score gate and the build, lint and test checks are the only brakes. For a repository with a green test suite, a strict linter and clear acceptance criteria in the issue, that can work. For a repository where the tests are thin, where the issue is a design question rather than a task, or where a change needs a domain decision, the loop will happily produce a passing build and a confident score for the wrong change, and it will merge it.

The second is that the loop is only as good as the issue text. The recommended workflow in the README starts with a PRD, then splitting it into fine-grained issues where each one can be completed in a single development session, with explicit acceptance criteria as checkboxes and dependencies marked. That splitting is the step where the human effort concentrates, and skipping it means the agents get a vague issue and score their own guess.

The third is the tool surface. Everything depends on external command line programs that this project does not ship, and the agent CLIs in particular move fast and change their output. The last push was on 2026-07-14 and the only release tag is v0.1.0 from 2026-04-27, so the project is young and its integration points are the fragile part.

There is a workable middle path for most teams. Run the tool in local issue mode, where it commits to a branch and does not push or merge, then review the diff yourself. You keep the iteration and scoring machinery and keep the merge decision, and the tool still tells you the score and the iteration count it reached. For a team that wants the automation but not the autonomy, that is the setting to start in.

Editorial conclusion

Adopt autoresearch if you want a repeatable loop that takes an issue, has agents implement and review it, and only opens a pull request when a quality gate is met, since that is the whole point of the tool. Do not adopt it if you need a workflow where a human approves every change, because the default path merges and closes the issue on its own. Before you run it, note that the tools it drives are separate command line programs, that the last push was on 2026-07-14 with the desktop release v0.1.0 from 2026-04-27, and that the primary language of the repository is Shell.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. Releases
5. smallnest/autoresearch on GitHub