autocontext: a self-improving harness for coding agents, with the receipts to prove it

The problem autocontext solves for agent builders

Most agent work happens in a loop that leaves nothing behind. You change a prompt or a tool description, run the task again, squint at the output, and keep whichever version feels better. The next person on the team starts from the same blank slate. autocontext is aimed at that gap. The README describes it as a harness for agent improvement: give it a goal, it runs the task against evaluation, keeps the useful lessons, discards dead ends, and leaves traces, reports, playbooks, datasets, and optional local-model training artifacts for the next run. The intended user is someone who already has an agent and a task, and wants the improvement to accumulate in files rather than in someone's memory. The repository topics name the runtimes it targets directly: claude-code, codex, hermes, pi, and pi-coding-agent.

How a run turns a goal into a scored generation

The mechanism is a generate, evaluate, select loop with a filesystem contract at the end. A run produces a directory keyed by run id. Inside it, trace.jsonl records the sequence of events, generations/<n>/ holds a strategy.json, an analysis.md and a score.json for each iteration, report.md summarises the run, and artifacts/ collects outputs. Separately, a knowledge directory keyed by scenario accumulates playbook.md, hints.md, a tools/ directory, and context_bundles/ with bundles, candidates, promotions and an active.json. That split matters. The run directory is evidence for one attempt; the knowledge directory is what survives it. Promotion into the live context is the interesting part. The README states that coach and architect context changes are stored as immutable candidates and are not served until matched candidate and incumbent trials confirm them, and that the live serving boundary can additionally require a cancellable independent audit and a durable campaign-wide false-promotion budget. In plain terms, a context change has to beat the thing currently in production under a controlled comparison before any agent sees it. The README also notes that exact causal credit is accepted only from verified single-component manifest additions, which is a deliberate refusal to attribute an improvement to a prompt edit just because both changed at once.

Installing autocontext and running a first solve

There are four install surfaces. The README lists a Python CLI, a Python library, a TypeScript or Node CLI, and a Pi extension. The PyPI package is named autocontext while the CLI it installs is named autoctx, which is the first thing that trips people up. The npm package is autoctx as well, and the README warns explicitly that it is not the unrelated autocontext npm package.

uv tool install autocontext==0.17.1

That installs the CLI. For the Node surface the README gives `bun add -g autoctx@0.17.4` and notes that the npm CLI and TUI require Node.js 22.19.0 or newer. The lowest-friction first run uses Pi, because according to the README it reuses your local agent auth rather than requiring a separate API key.

AUTOCONTEXT_AGENT_PROVIDER=pi \
AUTOCONTEXT_PI_COMMAND=pi \
autoctx solve "improve customer-support replies for billing disputes" --iterations 3

The goal is a plain-language string and --iterations bounds the loop. When it finishes, look at the run directory before changing anything. The README's layout puts a report.md at runs/<run_id>/report.md and the per-iteration scores at runs/<run_id>/generations/<n>/score.json. If the scores are flat across the three generations, the harness has not found anything to keep, and that is information about your evaluation, not about the tool. For the other providers, the README names anthropic, openai-compatible, openrouter, claude-cli, codex and pi-rpc. The .env.example sets AUTOCONTEXT_AGENT_PROVIDER=deterministic and AUTOCONTEXT_EXECUTOR_MODE=local by default, so an unconfigured checkout runs without any provider credentials at all.

Where the design gets awkward

The promotion machinery is the strongest idea here and also the source of the most friction. Requiring matched candidate and incumbent trials, plus in some configurations an independent audit and a campaign-wide false-promotion budget, means a context change can be rejected for reasons that have nothing to do with whether it reads better. That is the point, but it makes autocontext a poor fit for fast prompt iteration where you just want to try five phrasings this afternoon. The README is also candid about a narrower constraint: exact causal credit is accepted only from verified single-component manifest additions. Bundle two edits together and the harness will not tell you which one helped. On the operational side, the README states that Prime remote execution fails before provider creation when the configured pool cannot satisfy an accelerator request and never downgrades accelerator work to CPU. Failing early is the right call, but it means a misconfigured pool produces no run at all rather than a slow one. Finally, the repository has no retrieved releases, and the README documents version pins inline (0.17.1 for Python, 0.17.4 for npm, 0.10.1 for the Pi extension) rather than pointing at a release feed. The last push was on 2026-09-09, so the code is moving; the absence of a release history is a real cost for anyone who needs to pin and audit upgrades.

Self-hosted models and the constrained-output default

autocontext does not require a hosted provider. The README points to a self-hosted models guide covering the whole loop on vLLM, Ollama, or any OpenAI-compatible endpoint, and notes that constrained output matters more on open weights. That is a concrete claim about failure modes rather than a marketing line: a smaller model asked to emit an analysis document will drift off the expected shape, and the harness depends on structured role outputs to score and promote. The .env.example sets AUTOCONTEXT_CONSTRAINED_OUTPUT=true, which tells OpenAI-compatible role generation to use output schemas by default; setting it to false sends ordinary unconstrained requests and parses the returned Markdown instead. Two more variables describe the endpoint rather than the model: AUTOCONTEXT_PROVIDER_HOSTING can be set to local, and AUTOCONTEXT_PROVIDER_CAPABILITY takes fast, mid_tier or frontier. The comment in .env.example says an empty hosting value falls back to conservative transport inference, which is a quiet admission that the harness will assume the least capable case when you do not tell it otherwise. If you run a local endpoint, AUTOCONTEXT_LOCAL_MODEL fills every otherwise-unset role and tier slot, while explicit per-role or per-tier values take precedence over it. The roles are named in the file: competitor, analyst, coach, architect, translator and curator, and role-specific endpoints use matching <ROLE>_PROVIDER_* declarations.

How this differs from promptfoo and plain eval scripts

The closest familiar comparison is promptfoo, and the difference is in what gets compared. promptfoo evaluates prompts and model outputs against assertions and reports pass rates; it is a testing tool, and the artefact is a result you read. autocontext treats the context itself as the thing under test. Candidates are stored immutably, an incumbent is already serving, and a candidate only replaces it after a matched trial. The README also describes ablation-backed attribution, where controlled component trials let prompt selection demote low-value context rather than presenting edit-size correlation as causal. A hand-rolled eval script can do the same comparison, but it will not maintain a knowledge directory with playbook.md and hints.md across runs, and it will not carry the false-promotion budget. The trade-off is scope: promptfoo will tell you a prompt got worse, autocontext will tell you a prompt got worse and then refuse to ship it. If you only need the first answer, the second is overhead.

Licence, upgrade cost and what to check before adopting

autocontext is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant. The LICENSE file sits at the repository root. Nothing in the README suggests a separate commercial tier or a licence key, and the .env.example is entirely about providers and paths, not entitlements. That is the extent of what can be said here; this is not legal advice. On upgrade cost, the version pins are the practical concern. The README pins Python at 0.17.1, npm at 0.17.4 and the Pi extension at 0.10.1, and the Pi extension version is well behind the others, which suggests the surfaces do not move in lockstep. Anyone adopting this should decide which surface they are standardising on before pinning anything, because the Python and TypeScript packages carry separate version numbers. The .env.example also carries a compatibility note worth reading before you wire up credentials: AUTOCONTEXT_ANTHROPIC_API_KEY is described as a remaining compatibility alias, with provider-native env vars like ANTHROPIC_API_KEY preferred. There is a similar note that AUTOCONTEXT_PRIMEINTELLECT_API_BASE is compatibility-only because the installed SDK cannot safely receive a per-client custom base URL.

Editorial conclusion

Adopt autocontext if you already run agents against a task you can score and you want the improvement loop to be auditable rather than anecdotal; the filesystem-first run layout makes that possible. Skip it if you have no evaluation signal, because the harness has nothing to select on, or if you need a stable API surface, since the README describes an active research project with no releases. Before committing, install the pinned version with uv tool install autocontext==0.17.1, run the documented Pi quickstart for three iterations, and inspect runs/<run_id>/report.md and knowledge/<scenario>/playbook.md to confirm the promotion rules match how your team decides what a prompt change is worth.

Frequently asked questions

Official sources

1. greyhaven-ai/autocontext on GitHub
2. Issues
3. License: Apache-2.0
4. README