codex-autoresearch: an autonomous modify-verify-revert loop for Codex CLI

The problem is not code generation, it is the absence of a stopping rule

Most agent workflows assume you want something written. This skill assumes you want a number moved. The README frames it plainly: tell Codex what measurable result you want, and it inspects the repository, confirms the experiment with you, changes one thing, verifies it, keeps improvements, reverts failures, and repeats until the target is reached. The named candidates are test failures, coverage, type errors, warnings, latency, binary size and reproducible security findings, with an open clause for any other outcome a command can measure. That list is the honest boundary of the tool. If your goal cannot be reduced to a command that exits successfully and prints one finite number on its final non-empty stdout line, the skill has nothing to iterate against. The audience is therefore narrow and specific: engineers with an existing test or benchmark harness who want a machine to grind through the long tail of small fixes rather than a machine to design the fix. The loop is credited in the repository description as inspired by Karpathy's autoresearch concept, and the shape follows that idea: many cheap trials, hard discard of anything that does not improve the measurement.

A control script owns Git, Codex owns the hypothesis

The division of labour is the most interesting design decision in the project. According to the README, the control script owns commits, verification, rollback and state, while Codex owns the hypotheses and code changes. That split means the agent never decides whether a trial succeeded. It proposes a change; the script commits it, runs the metric, compares the number, and either keeps the commit or issues a git revert. The loop diagram in the README runs: inspect evidence, change one focused thing, commit and measure, then branch on improved plus guard passes leading to keep, otherwise revert, then append an audit event, then repeat until target. Two modes exist. In foreground the run lives inside the current Codex task and continuation is handled by an official Codex Goal, with pause and resume available through that Goal. In background the run is a detached controller that spawns one codex exec worker per iteration. Both modes use the same experiment rules, and a run uses one mode at a time. Foreground suits watching and steering live; background suits long or overnight runs, where you query the skill for status, stop or resume. The state model is deliberately unforgiving. events.jsonl is described as the state history, and missing, malformed, contradictory or partial state is treated as an error, with the skill never guessing a result from old files or conversational memory.

Install, invoke, confirm: the actual commands

Installation goes through the Codex skill installer rather than a package manager. The README gives the invocation as $skill-installer install https://github.com/leo-lilinxiao/codex-autoresearch. The quick start then opens a clean Git repository with Full Access using codex --dangerously-bypass-approvals-and-sandbox, and the skill is invoked with $codex-autoresearch followed by the goal in plain language, for example: Reduce `python3 scripts/score.py` error_count to 0. Before the first write, Codex shows a confirmation block containing the goal and numeric target, the repository-relative paths it may change, the metric command and explicit parser, an optional regression guard, foreground or background mode, and an optional iteration limit. The README's sample confirmation reads Baseline: 5, Target: 0 (lower is better), Scope: src/, Verify: python3 scripts/score.py with JSON key error_count, Guard: python3 -m pytest -q, and then asks whether to run in the foreground or background. Initialization requires a clean named Git branch, and one run manages one repository. The README states that no Codex configuration changes or special prompt syntax are required, and a manual or development install path is documented separately in docs/INSTALL.md. The flag in the launch command is worth reading twice: the skill expects an environment where Codex is not stopping for approvals.

The metric contract is where most setups will fail

The verify command must exit successfully and place one finite number on its final non-empty stdout line. It may instead print a JSON object on that line when Codex names one numeric key explicitly. The README shows two accepted shapes: a bare 7, or {"error_count": 7, "passed": 12} with error_count declared as the key. That is a tight contract and it is the part most likely to bite. A script that prints a progress bar after its result, or writes the number to stderr, or emits a human-readable summary line last, does not conform. The escape hatch is the guard, described as being for behaviour the metric does not protect, such as a test suite around a latency benchmark. The guard must pass at baseline, which is a sensible gate: if your test suite is already red, the run cannot start. Notice what the guard does not do. It is a pass or fail check, not a second objective. There is no Pareto logic in the material, and the release note for v0.4.0 mentions multi-metric acceptance, but the README's own contract still describes one number per verify command. If your quality bar genuinely needs two numbers traded against each other, the documented mechanism gives you one objective plus one boolean, and you should treat that as the design rather than a gap to be worked around.

What lands in autoresearch-results, and what stays out of Git

Run artifacts live in autoresearch-results/ and stay uncommitted. The README lists run.json as the immutable confirmed configuration, events.jsonl as the append-only baseline, iteration, stop and completion history, logs/ for full metric, guard and background worker output, runtime.json for background process state, runtime.log for background controller lifecycle events, and report.html as an optional regenerated visual snapshot. The README is explicit that the HTML report is a replaceable snapshot, not runtime state, which is the right call: a rendered file that drifts from events.jsonl should never be the thing you trust. Review is done through the skill itself. Asking for $codex-autoresearch show experiment history prints a table with sequence, iteration, event, previous value, trial value, retained value and a description, plus run id, status, mode, metric trajectory and target. The sample output shows a run going from baseline 2 to target 0 across three iterations, with one discard at trial 3 and two keeps. The same validated events can be exported as TSV or rendered to report.html through $codex-autoresearch export experiment history as TSV and $codex-autoresearch generate an HTML report. Because the artifacts are never staged, your repository history stays clean of run bookkeeping, at the cost of the results directory being local to wherever the run happened.

The safety model is strict enough to be annoying, on purpose

Every trial is a Git commit. A non-improving trial or a failed guard is reverted with git revert. Out-of-scope edits, branch changes, HEAD drift, malformed metrics, command failures, timeouts and generated byproducts stop the run with an exact error and log path. A run reports complete only when the retained metric reaches the confirmed target. A genuine external blocker is reported explicitly, and the README adds the distinction that matters: a difficult or unsuccessful hypothesis is not treated as blocked. The README defends this directly, stating that silent recovery makes long autonomous runs impossible to trust. That defence is correct and it also defines the failure mode. Any tool that writes a build artifact into a tracked path, any hook that touches HEAD, any metric script that is flaky under load, will halt the loop rather than be tolerated. For an overnight background run this is the difference between a stopped run you can inspect and a run that quietly wandered. It also means the skill is a poor fit for repositories where side effects are normal, such as anything that regenerates lockfiles, snapshots or fixtures as part of a build.

Compared with a plain agent session, the difference is the referee

The obvious alternative is to run Codex directly and ask it to fix the failing tests, iterating in conversation. That approach has no independent referee. The model both proposes the change and judges whether it worked, and the judgement lives in the context window rather than in a file. codex-autoresearch moves the judgement into a shell command and the record into events.jsonl, and it makes the revert mechanical rather than a suggestion. A second alternative is a conventional CI-driven optimisation loop, a script you write yourself that runs a benchmark, mutates a parameter, and keeps the best result. That shares the mechanical revert but not the hypothesis generation: a hand-written loop searches a parameter space you defined in advance, while this skill asks the model to read the repository and propose the next change. The trade is that your hand-written loop knows exactly what it is allowed to touch, and this one relies on the scope paths confirmed at the start plus the out-of-scope detection to enforce the same boundary after the fact. If your search space is already enumerable, the script you write yourself is cheaper and easier to reason about. The skill earns its place when you do not know what the next fix is.

Licence, maintenance and what the release cadence tells you

The project is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are preserved. That is a permissive default and it is the only licence statement in the supplied material; nothing here addresses trademark, patent or contributor terms, and this is not legal advice. On maintenance, the supplied facts show three releases in roughly six weeks: v0.4.0 on 2026-04-24 covering workspace-owned results and multi-metric acceptance, v0.5.0 on 2026-05-13 covering Codex Goals and reliable hooks, and v0.6.0 on 2026-06-02 covering Codex CLI readiness and runtime stability. The last push to the default branch is 2026-07-13, and the repository is not archived. Read the version titles rather than the version numbers. Two of the three releases are about runtime plumbing and Codex integration rather than new search behaviour, which suggests the project is still tracking a moving host, the Codex CLI, and that the surface most likely to break on upgrade is the hook and Goal integration rather than the metric contract. The cost of an upgrade is therefore not in your metric script. It is in re-checking that your Codex CLI version still supports the Goal-based foreground continuation and the background worker invocation the skill depends on, and in re-running the installation step so the skill definition matches the CLI you have.

Editorial conclusion

Adopt it if you have a repository whose progress can be expressed as one finite number on the last stdout line of a command, a test suite you can attach as a guard, and a tolerance for runs that stop on the first out-of-scope edit. Do not adopt it if your goal is qualitative, if your metric needs several numbers to be meaningful, or if you cannot keep a clean named Git branch for the duration. Verify first that the metric command exits zero and prints exactly one number, that the guard passes at baseline, and that your Codex CLI version matches the v0.6.0 readiness work, because a malformed metric stops the run rather than being interpreted.

Official sources

1. Issues
2. leo-lilinxiao/codex-autoresearch on GitHub
3. License: MIT
4. README
5. Releases