auto-harness: letting a coding agent rewrite its own harness overnight

The idea

The opening line of the README is the whole proposal in one sentence: give a coding agent a benchmark and an agent file, and let it iterate overnight. It reads failures, improves the system prompt and tools, gates every change against a self-maintained eval suite, and repeats.

The name of the pattern is the interesting part. The agent being improved is not being trained. Its weights do not change. What changes is agent/agent.py, the file that holds the system prompt and the tool definitions, and the agent is rewriting it based on what it observed going wrong.

That distinction matters because it makes the loop tractable. Prompt and tool changes are reviewable text, they take effect immediately, and they can be rolled back. Fine-tuning a model on the same failure set is a much heavier operation with a much worse feedback loop. What the framework does is set up a repeatable loop over the cheap kind of change.

The demonstration is stated in the README's own terms. This repo is a simplified version of the authors' auto-harness agent setup, and they demonstrate the system on Tau3 benchmark tasks where the agent's score improves from 0.56 to 0.78, roughly a 40 percent jump, while mining failures and automatically maintaining live evals. A full write-up lives on the company's blog.

The primary language is Python, the licence is MIT, the default branch is main, and the homepage is the blog post describing the approach. The last push was on 2026-09-16 with the project not archived, and the repository publishes no GitHub releases.

The loop itself is defined in PROGRAM.md, and the coding agent edits agent/agent.py to improve the agent and appends findings to workspace/learnings.md after each iteration. So the instructions for the optimisation are themselves a file the framework writes, copied from a per-benchmark template, and the agent's accumulated notes live in a separate learnings file that survives between iterations.

The loop and the files that implement it

The How it works section gives the cycle as a single line:

run benchmark → analyze → improve agent/agent.py → gate → record → update learnings → repeat

Seven files implement it, and the naming makes the structure obvious.

agent/agent.py is the agent being optimised, copied from a benchmark-specific template. agent/templates/ holds the starting-point templates for each benchmark and is read-only. benchmark.py runs your benchmark and returns per-task rewards. gating.py is a three-step gate consisting of an eval suite, a full test val_score, and suite promotion. record.py appends iteration results to workspace/results.tsv. prepare.py sets up the workspace, copies the templates and runs the baseline. program_templates/ holds benchmark-specific PROGRAM.md instructions, and PROGRAM.md itself is the instructions the coding agent follows, copied from that template by prepare.py.

The division of labour is worth reading carefully. The thing being optimised is a single file. The evaluation is separated out into gating.py. The result log is separated out into record.py. And the instructions to the optimiser live in a markdown file that a human can read and edit, which means the loop's behaviour is steerable without touching code.

The three-step gate is the part that makes this safe to run unattended. The pipeline line reads run benchmark, analyze, improve agent/agent.py, gate, record, update learnings, repeat, and the gate sits between the improvement and the recording. So a change that does not pass the gate is not accepted, and the agent has to go round again.

The three components of the gate are different in kind. The eval suite is a set of checks, derived from the mined failures, which is the self-maintaining part: as new failure modes are found, new checks are added. The full test val_score is the objective measure, running the whole benchmark rather than a subset. And suite promotion is the step where a passing check from the suite graduates into the permanent set, which is the mechanism that keeps the suite from growing without limit while still learning from each failure.

So the framework has an answer to the obvious objection. If you let an agent rewrite itself, it can overfit to the benchmark or quietly weaken a test it dislikes. The gate plus the promotion mechanism is the answer, and it is the piece to look at when judging whether the score improvement is real.

The supported benchmarks

Three benchmarks are supported, and the table is worth reading because the diversity is the point.

tau-bench covers customer service in retail, airline and telecom, with 114 retail tasks, 50 airline and 114 telecom, and the agent interface is structured tool calls via tau2. Terminal-Bench 2.0 covers real-world terminal tasks in coding, sysadmin and security, with 89 tasks and the interface is bash commands via Harbor containers. And BIRD-Interact is interactive text-to-SQL, multi-turn with CRUD over Postgres, with 300 tasks in the lite version and 600 in the full one, and the interface is a Google ADK agent against a three-service environment consisting of a user simulator, a database environment and a system agent.

Three very different shapes. The first is a function-calling API in a customer service domain, where the difficulty is policy compliance and multi-turn conversation. The second is a shell in a container, where the difficulty is that the agent has to actually operate a machine. The third is a database with a simulated user in the loop, where the difficulty is generating correct SQL across a conversation.

That spread matters for the self-improvement claim. A harness that only works on tool-calling benchmarks would tell you little. Running the same loop over a shell environment and a SQL environment tests whether the gate and the mining machinery generalise, and the framework's design does not obviously depend on any of the three domains. The scripts in the repository are per-benchmark templates, which is the right place for that variation.

The BIRD-Interact entry has one detail that is unusual for a benchmark harness. The agent interface is not just one agent. It is a Google ADK agent against a three-service environment, so there is a user simulator and a database environment alongside the system agent being optimised. That means the evaluation involves a simulated user, which is a harder thing to make deterministic, and it is why the setup for that benchmark is the longest of the three.

The topics on the repository are agent-framework, agent-orchestration, llm-application and machine-learning, with python as the language.

Running it, and the costs you inherit

The first-run checklist is unusually thorough, and it reads as a list of things that will bite you.

The first item is to choose one benchmark and install its prerequisites. Terminal-Bench 2.0 needs the harbor CLI and a configured sandbox provider, or a running Docker daemon if you set env_provider to docker. BIRD-Interact needs Python 3.12 or newer, Docker, git-lfs and access to the ground truth data. And tau-bench uses Docker Compose and the image built by docker compose build.

The second item is configuring the experiment. You copy experiment_config.yaml.template to experiment_config.yaml, uncomment only the benchmark section you intend to use, and review its models, dataset or domain, and max_concurrency. The template is entirely commented out, so copying it alone does not select a benchmark. That is a small but important detail, because a fully commented config means the failure mode is a confusing error rather than an obvious one.

The third is credentials, and the fourth is the one people miss. Docker Compose loads .env through env_file, but when you run python prepare.py directly, prepare.py does not load .env automatically, so you have to export the variables in your shell first. The README suggests set -a; source .env; set +a from the repository root for Bash and Zsh users who have reviewed the file.

The fifth is the cost warning, and it is the most important line in the checklist. On a fresh workspace, python prepare.py performs setup and launches the baseline benchmark. This makes model API calls and may incur sandbox-provider charges. Review your provider budget before running it, because it is not a setup-only check. And after initialization completes, optimisation and gating run additional benchmark tasks and can incur further usage charges.

That is a distinction people get wrong. Running the setup command looks like preparation, but it runs 89 tasks for Terminal-Bench or 300 for BIRD-Interact, and each of those is a full agent interaction against a model. Then the loop itself runs the benchmark again, and again, and again, overnight.

The Terminal-Bench quick start shows the full sequence:

# 1. Clone the repo
git clone https://github.com/neosigmaai/auto-harness
cd auto-harness

# 2. Install harbor
uv tool install harbor

# 3. Set up environment variables
cp .env.example .env
# edit .env — set OPENAI_API_KEY, plus your sandbox provider's credential
# (E2B_API_KEY, DAYTONA_API_KEY, or MODAL_TOKEN_ID + MODAL_TOKEN_SECRET) —
# not needed if you're using env_provider: "docker"

# 4. Configure the experiment
cp experiment_config.yaml.template experiment_config.yaml
# edit experiment_config.yaml — uncomment the terminal-bench section

# 5. Initialize workspace + run baseline (runs all 89 tasks, generates train/test split)
python prepare.py

# 6. Start the optimization loop
# Point your coding agent at the repo and prompt:
#   "Read PROGRAM.md and start the optimization loop."

The BIRD-Interact one is the same shape with a longer preparation phase:

# 1. Clone this repo
git clone https://github.com/neosigmaai/auto-harness
cd auto-harness

# 2. Set up environment variables
cp .env.example .env
# edit .env — set OPENAI_API_KEY (or ANTHROPIC_API_KEY)

# 3. Configure the experiment
cp experiment_config.yaml.template experiment_config.yaml
# edit experiment_config.yaml — uncomment the BIRD-INTERACT section

# 4. Initialize — prepare.py auto-provisions everything:
#      - clones BIRD-Interact-ADK into ./bird_interact_adk/ (gitignored)
#      - creates an isolated .venv-adk with the ADK's deps
#      - clones the bird-interact-lite dataset from HuggingFace
#      - starts the Postgres Docker container
#      - runs the baseline (300 tasks) and generates the train/test split
python prepare.py

# 5. Start the optimization loop
# Point your coding agent at the repo and prompt:
#   "Read PROGRAM.md and start the optimization loop."

And the ground truth step for BIRD-Interact is a one-time manual action, because the public dataset ships without gold SQL to prevent data leakage. You email the address given with the exact subject line, run the merge script shown by prepare.py with the jsonl you receive, and re-run prepare.py:

1. Email `[email protected]` with subject `[bird-interact-lite GT&Test Cases]`
2. Run the `combine_public_with_gt.py` script shown by prepare.py, using the jsonl you receive
3. Re-run `python prepare.py`

The data leakage reasoning is the interesting part. The benchmark withholds the answers so that a system cannot be scored against a test set it has already seen, and the setup detects the absence and tells you exactly what to do about it.

The repository, and the container layout

The top-level listing is small: .env.example, .gitignore, a Dockerfile, LICENSE, PROGRAM.md, README.md, an agent/ directory, benchmark.py, docker-compose.yml, experiment_config.yaml.template, gating.py, prepare.py, program_templates/, pyproject.toml and record.py.

The pyproject.toml is interesting because of one line. The dependencies are openai, pyyaml and tau2, and the tool.uv.sources section pins tau2 to a git repository at a specific revision, the sierra-research/tau2-bench repository. So the benchmark itself is a git dependency rather than a published package, pinned to one commit.

That is a normal choice for a fast-moving research dependency, and it cuts both ways. It means the harness runs against exactly the benchmark version the authors used, which is what you want for reproducing their numbers. It also means a git fetch during install, and that the pinned revision has to stay reachable. The Dockerfile confirms the dependency is fetched over git, since it installs git alongside uv specifically so that uv can fetch tau2 from git.

The Dockerfile is a Python 3.12 slim base with git and uv installed, the dependency manifest copied first for better layer caching, and then uv sync --no-dev. The caching comment is a detail that shows care: copying pyproject.toml before the source means the dependency layer is reused unless the manifest changes.

The docker-compose file is where the editable workflow becomes clear. The service is called autoeval, it builds from the repository, and it loads .env through env_file. The environment section sets AGENT_MODEL with a default of gpt-5.4 and TAU2_DATA_DIR to /tau2_data.

The volumes are the important part. The agent directory, the workspace directory, the experiment config and each of the four Python modules are mounted into the container, with a comment explaining why: mount all editable files so changes take effect without rebuilding. So the agent can edit agent/agent.py, the result is written to workspace/results.tsv on the host, and the container does not need to be restarted or rebuilt between iterations.

There is one subtlety in the data directory mount that the compose file spells out in a comment. TAU2_DATA_DIR has two layers: the host-side substitution here controls which directory is mounted, while the container always sees it as /tau2_data, which is set separately in the environment block. To use a different data directory you set TAU2_DATA_DIR in .env.

Two final touches: stdin_open and tty are both true, which is what lets an interactive coding agent drive the container and see its output, and the default agent model in the compose file is a specific frontier model, so the harness is configured to run against a capable model out of the box.

Editorial conclusion

Adopt auto-harness if you are building an agent and want to try letting it rewrite its own prompt and tool definitions against a benchmark overnight, since the loop, the gate and the per-benchmark templates are all laid out and the reported result is a jump from 0.56 to 0.78 on tau-bench. Do not adopt it without reading the gate, because the whole value of the approach rests on a change failing to be accepted when it does not improve the held-out score, and the self-maintained eval suite is the part most likely to be overfit. Before you commit, budget for the API and sandbox charges, since even the preparation command runs the full baseline rather than only configuring things, export the environment variables yourself when running prepare.py directly because it does not load .env, and remember that the tau2 benchmark is pinned to a git revision, so the dependency is fetched from source rather than from a package registry.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. neosigmaai/auto-harness on GitHub
4. Project website
5. README