# Agentic Harness Engineering: evolving a coding agent harness with observability

> AHE is a Python framework that keeps a base model fixed and rewrites the harness around it, the system prompt, tool descriptions, middleware, skills and memory, using traces as the feedback signal. It suits teams with a Terminal-Bench style evaluation loop and the budget to run many sandboxed rollouts, and it does not suit a team with no way to score a run automatically.

**china-qijizhifeng/agentic-harness-engineering** — Official AHE code — Agentic Harness Engineering: observability-driven automatic evolution of coding-agent harnesses (concurrent w/ meta-harness). NexAU-AHE reaches 84.7% ± 2.1 pass@1 on Terminal-Bench 2 (GPT-5.5). Lifts GPT-5.4 69.7→77.0% over 10 iters, beats Codex/ACE/Training-Free GRPO; frozen harness transfers to SWE-bench-Verified.

- Repository: https://github.com/china-qijizhifeng/agentic-harness-engineering
- Website: https://arxiv.org/abs/2604.25850
- Stars: 963 · Forks: 106
- Language: Python
- License: MIT
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/china-qijizhifeng-agentic-harness-engineering

## The harness moves, the model does not

Agentic Harness Engineering, abbreviated AHE, is described in the documentation as an open observability system for automatically evolving the harness around a coding agent. The framing matters more than the name suggests. The base model is held fixed. What the framework changes are the harness components: system prompts, tool descriptions, tool implementations, middleware, skills, sub-agents and long-term memory.

That split targets a specific audience. If you are deciding which model to buy, this project is not about that. It is about the layer most teams write by hand and then never revisit, the scaffolding around the model, and it treats that scaffolding as an optimisable artefact with a measurement attached to it. The repository is the official code for the accompanying paper, with the paper on arXiv, a PDF in the repository root, a Chinese README alongside the English one, and a blog post on Dawning Road that gives a longer account of the exploration.

The implementation is Python, requires Python 3.13 or newer, and is managed with uv. The pyproject.toml declares the project as agentic-harness-engineering version 0.1.0, exposes a console entry point named agentic-harness-engineering that points at evolve:main, and pins the constraint claude-agent-sdk<0.1.49.

## Three observability layers feed one loop

The documentation describes the system as resting on three observability layers, and this is the part worth reading carefully because it explains where the evidence for an edit comes from.

Component observability comes from NexAU, which decomposes the harness into seven orthogonal, file-level components, each git-tracked so every edit is auditable and revertible. The file level part is the load bearing detail: an edit is a diff, not a database row, so a bad iteration can be undone by reverting files.

Experience observability comes from Agent Debugger, which distils roughly ten million tokens of raw traces into layered, sourced reports. The optimiser reads digests by default but can drill back to any rollout's raw trace, which means a summary is a navigation aid rather than a boundary.

Decision observability comes from Evolve Agent, which proposes evidence-backed edits and predicts their impact, and is automatically falsified by the next iteration's flipped tasks. The loop itself is described as ten evaluate, analyse, improve iterations.

## Getting the loop running with uv, tmux and three keys

The prerequisites are Python 3.13 or newer, uv and tmux, with install commands given per platform:

```bash
# macOS
brew install uv tmux

# Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
sudo apt install -y tmux
```

Then clone and sync:

```bash
git clone https://github.com/Curry09/agentic-harness-engineering.git
cd agentic-harness-engineering
uv sync
```

uv sync installs every dependency declared in pyproject.toml, and that file pulls in pyyaml, python-dotenv, the e2b SDK, NexAU pinned to the v0.3.9 tag, a harbor fork from git+https://github.com/Curry09/harbor-LJH.git, and agent_debugger_core, which is not fetched from an index but supplied from a local path inside the tree at agents/evolve_agent/skills/agent-debugger-cli/_source.

Configuration is a copy of the template:

```bash
cp .env.example .env
```

At minimum you need LLM_API_KEY and LLM_BASE_URL for the main endpoint, E2B_API_KEY for the sandbox and SERPER_API_KEY for the web search used by evolve_agent. GITHUB_TOKEN is required by the environment template as well. ADB_LLM_* and GPT54_LLM_* are optional and fall back to the main credentials if left unset.

## E2B sandboxes carry a concurrency cap you must check first

Every rollout runs inside an E2B sandbox, and the deployment notes are unusually clear about the trap here. In SaaS mode you set only E2B_API_KEY and leave E2B_API_URL and E2B_DOMAIN unset, and the SDK talks to e2b.dev automatically. SaaS E2B enforces a per-account concurrent sandbox limit tied to your tier. If the run tries to spawn more sandboxes than the cap allows, the extra sandboxes fail to start and the iteration stalls.

The self-hosted route avoids that account cap but not the underlying limit, since the cluster's hardware capacity still applies. The variables to set are:

```dotenv
E2B_API_KEY="your_e2b_key"
E2B_API_URL="https://your-e2b-host.example.com"
E2B_DOMAIN="your-e2b-host.example.com"
```

This is the operational constraint most likely to shape a first run. Ten iterations of parallel rollouts against a tier-limited SaaS account can stall on quota rather than on anything to do with the harness. The notes advise checking the tier quota and staying safely under it before raising parallelism in the harbor or experiment config, and the environment template exposes E2B_SANDBOX_TIMEOUT=3600 along with commented retry knobs E2B_MAX_RETRIES and E2B_RETRY_MAX_WAIT.

## Templates are built per task before the first rollout

A rollout does not start from a bare container. It starts from a prebuilt template that already has uv and the NexAU/harbor virtual environment at /opt/nexau-venv, and those templates are built once per dataset:

```bash
# Build every template declared by the dataset, 16 in parallel
uv run python scripts/build_templates.py --dataset-dir /path/to/dataset -j 16

# Resume after a failure: only retry tasks whose latest E2B build status is ERROR
uv run python scripts/build_templates.py --dataset-dir /path/to/dataset --retry-failed

# Build a specific subset of tasks
uv run python scripts/build_templates.py --dataset-dir /path/to/dataset task_a task_b
```

The dataset is a pack from laude-institute/harbor-datasets, and you point --dataset-dir at the subset you need. The directory must contain one subdirectory per task, and each task needs a task.toml declaring [environment].docker_image, with an environment/Dockerfile as a fallback. The alias for a template is the task name with dots replaced by hyphens. The default packages baked into each template come from a constant named DEFAULT_NEXAU_PACKAGES in scripts/build_templates.py, which deliberately differs from the host-side harbor-LJH in pyproject.toml, and you can override that list with --nexau-package.

That mismatch between the in-sandbox package set and the host-side one is a design choice rather than an accident, and it is the kind of detail that saves an afternoon when a version behaves differently inside the sandbox than on the host.

## What the reported numbers claim and what they leave open

The project reports specific results, and it is worth reading them with the harness's own framing in mind. AHE on GPT-5.4 lifts Terminal-Bench 2 pass@1 from 69.7% to 77.0% over ten evaluate, analyse, improve iterations, which the same section compares against a hand-written Codex baseline at 71.9% and against the self-evolving ACE and TF-GRPO baselines. The project description adds that AHE on GPT-5.5 reaches 84.7% plus or minus 2.1 pass@1, and the news section records a third place ranking on the Terminal-Bench 2.0 leaderboard as of 2026-05-15.

The transfer result is the more interesting claim for an adopter. The project states that the frozen harness transfers without re-evolution to SWE-bench-verified and to four alternate base models, and reads that as evidence the evolved components encode general engineering experience rather than benchmark-specific tuning.

What is not established is just as relevant. The scores are benchmark numbers, produced by the authors' own harness and their own evaluation setup, and the repository ships no third-party replication. The evolved components are also only as good as the task set you point --dataset-dir at. A different dataset means different templates, different traces and a different set of proposed edits, so the headline figures do not transfer to your workload by themselves.

## No tagged releases, and an environment file full of credentials

The repository is not archived and the last push was on 2026-08-03, so the code is moving, but there are no GitHub releases and no tags to install. The version marker lives in pyproject.toml as 0.1.0, and the project is published as a private-style application package rather than a library. Anyone adopting this is tracking a moving main branch, which means the usual upgrade question becomes a re-read of the diff rather than a version bump.

Licensing is MIT, stated in pyproject.toml as license = { text = "MIT" } and shown as a badge in the project header, so the code can be modified and shipped inside a commercial product with attribution only.

The credential surface deserves attention. The .env.example template asks for a primary LLM key and base URL, an optional debugger-specific key, an optional gpt-5.4 experiment key, an E2B key with optional self-hosted URL and domain, a GitHub personal access token, and a Serper API key for web search. Langfuse keys, an HTML parser URL with its key and secret, and a Feishu webhook are optional observability and convenience hooks. The template notes that the .env file is in .gitignore and that the three BP_HTML_PARSER variables must all be present for the enhanced fetch path to enable, otherwise it degrades to plain fetching.

## Conclusion

Adopt Agentic Harness Engineering if you already run a repeatable agent evaluation, if you can afford sandboxed rollouts at the pace the loop requires, and if your harness lives in files you can diff. Skip it for a one-off agent build, for a setup without a task set that scores automatically, and for anyone unwilling to supply an LLM endpoint, an E2B account and a GitHub token. Verify first that your dataset subset has one subdirectory per task with a task.toml declaring [environment].docker_image, then run the template build with --retry-failed before you raise parallelism.

## FAQ

### What does Agentic Harness Engineering actually change in an agent?

It holds the base model fixed and evolves the harness components around it: system prompts, tool descriptions, tool implementations, middleware, skills, sub-agents and long-term memory. Those components are file-level and git-tracked so each edit stays auditable and revertible.

### What do I need before running Agentic Harness Engineering?

Python 3.13 or newer, uv, tmux, and environment variables for an LLM endpoint, an E2B sandbox, web search and GitHub access. Copy .env.example to .env and set at least LLM_API_KEY, LLM_BASE_URL, E2B_API_KEY and SERPER_API_KEY.

### Why does my Agentic Harness Engineering run stall during an iteration?

The documentation flags this as the concurrency cap on SaaS E2B: extra sandboxes beyond the per-account tier limit fail to start and the iteration stalls. Check the tier quota and stay under it, or point E2B_API_URL and E2B_DOMAIN at a self-hosted cluster.

## Sources

- [china-qijizhifeng/agentic-harness-engineering on GitHub](https://github.com/china-qijizhifeng/agentic-harness-engineering)
- [Issues](https://github.com/china-qijizhifeng/agentic-harness-engineering/issues)
- [License: MIT](https://github.com/china-qijizhifeng/agentic-harness-engineering/blob/main/LICENSE)
- [Project website](https://arxiv.org/abs/2604.25850)
- [README](https://github.com/china-qijizhifeng/agentic-harness-engineering/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/china-qijizhifeng-agentic-harness-engineering