DeepSWE: running the 113-task coding agent benchmark with Pier

The problem DeepSWE targets: agents that pass tests they wrote themselves

Most coding-agent evaluations hand the agent a repository, let it run the test suite, and count green. That loop rewards an agent for writing tests that match its own implementation. DeepSWE's answer is to split the two. The agent works inside an isolated environment and commits its work when it is done. A [[verifier.collect]] hook in each task.toml extracts those commits as a patch. That patch is then applied and graded in a pristine container, so nothing the agent left behind in its working tree, no installed dependency and no edited test file, travels into the grading step. The README states the reference patch in solution/ is never used at grading time; it exists so reviewers can spot-check correctness offline. That is the design decision worth noticing. The benchmark treats the solution directory as human documentation, not as an answer key the grader consults.

The audience is anyone who has to say something defensible about how well a coding agent handles work that spans more than one file. The corpus is 113 tasks across TypeScript, Go, Python, JavaScript and Rust, drawn from active open-source repositories. Long-horizon is the operative word: these are not LeetCode prompts dressed up with a repository wrapper.

How a DeepSWE task is laid out and what the verifier actually checks

Each task follows the Harbor task format, and the repository layout is small enough to read in one pass. task.toml carries metadata: repo, base commit, language, image, limits. instruction.md is the prompt the agent sees. environment/ holds a Dockerfile that reproduces the prebuilt image. tests/ holds the verifier entry point, the held-out tests and the grader config. solution/ holds the reference solution, held out from the agent.

The grading contract is behavioral. According to the README, the verifier exercises the behavior the prompt describes and accepts any solution whose observable behavior is correct, regardless of internal symbol names or structure. That matters if you plan to compare agents across languages: an agent that renames a helper or restructures a module is not penalized for it. The cost is that the verifier has to be written carefully, because a behavior-based check that is too loose will pass wrong implementations and one that is too tight will reject correct ones. The README does not describe how task authors validated that boundary.

Each run produces a directory of artifacts rather than a single number. reward.json holds structured scores, a binary reward plus pass fractions. ctrf.json is a machine-readable test report with failure messages. test-stdout.txt holds raw suite output and a list of failure reasons. run.log captures raw stdout and stderr. reports/ holds framework-native report and log files from the grader. If you only read reward.json you will miss why a run failed; test-stdout.txt and ctrf.json are where the diagnostic work happens.

Installing Pier and running your first DeepSWE task

DeepSWE does not ship its own runner. The README points at Pier, a Harbor-compatible framework for sandboxed coding-agent evals, and the quickstart is four commands. Clone the benchmark repository, install Pier as a uv tool, export the API key for whichever provider you are testing, and run.

git clone https://github.com/datacurve-ai/deep-swe
uv tool install datacurve-pier

# Claude Opus 4.8
export ANTHROPIC_API_KEY=...
pier run -p deep-swe/tasks --agent mini-swe-agent --model anthropic/claude-opus-4-8

# GPT-5.5
export OPENAI_API_KEY=...
pier run -p deep-swe/tasks --agent mini-swe-agent --model openai/gpt-5.5

The -p flag points at the task directory, --agent selects the harness and --model selects the model. Running the full corpus is the default when you pass deep-swe/tasks without a task count. Expect the run to be long and to consume provider quota; 113 tasks with an agent loop are not a quick check.

For a cheaper pass, sample deterministically. The README describes this as a deterministic random subset of the 113-task corpus, which means the same seed gives the same tasks on every machine, so two people comparing notes are comparing the same work.

pier run -p deep-swe/tasks --agent mini-swe-agent --n-tasks 10 --sample-seed 0

To iterate on a single task, point -p at the task directory itself. The README does not give an example task id, so substitute the directory name you see under tasks/.

pier run -p deep-swe/tasks/<task-id> --agent mini-swe-agent

Pass --env modal if you want parallel sandboxes on Modal. mini-swe-agent is model-agnostic, and Pier also drives claude-code, codex, gemini-cli and opencode directly.

The v1.1 grading change and the Pier version floor it creates

Since v1.1, DeepSWE grading uses Harbor's separate verifier environment rather than a shared one, and the README states this requires Pier newer than 0.3.0. This is the single most likely way to get a wrong result without noticing. An older Pier will still run tasks; it just will not grade them the way the published scores were produced. The README does not document a version check command or a warning emitted at runtime, so the version floor is something you enforce yourself before trusting a number.

The same section explains why Pier exists at all. Pier began as a fork of Harbor to support CLI agents in air-gapped tasks: Harbor blocks all outbound traffic in allow_internet = false tasks, including dependency installs and LLM API calls. Pier adds per-agent network allowlists, giving an agent only the network access it needs while keeping the task environment isolated. For a benchmark whose whole premise is isolation, that is a load-bearing change rather than a convenience. Pier also adds more complete trajectory metadata, a trajectory viewer, and pier critique run for analyzing agent trajectories. The README states all leaderboard scores were produced with Pier running mini-swe-agent on Modal.

Where DeepSWE is the wrong tool

The benchmark is expensive by construction. Every task carries a Dockerfile and runs in a sandbox; grading happens in a separate pristine container. If you want a fast signal on whether a model can write a correct function, this is the wrong instrument, and a smaller suite will tell you more per dollar. The 113 tasks are also drawn from real repositories, which means the difficulty is uneven in a way that a synthetic suite is not: some tasks will hinge on understanding an unfamiliar codebase, others on a narrow behavioral fix.

There is a second, sharper limitation. Because the verifier accepts any solution with correct observable behavior, the score measures whether the agent produced the right behavior, not whether it produced maintainable code. An agent that satisfies the held-out tests with an ugly patch scores the same as one that refactors cleanly. For teams evaluating agents as code reviewers or long-term contributors, that gap is the whole question, and DeepSWE does not answer it. The README also does not document rollback, resumption or partial-run recovery, so a run that dies halfway is not something the documented interface helps you salvage.

DeepSWE versus SWE-bench-style evaluation

The obvious comparison is SWE-bench and the family of benchmarks that followed it, which also draw tasks from real repositories and grade against held-out tests. The difference in approach is where the agent's work is turned into a patch. SWE-bench-style harnesses typically take a diff from the agent's working environment and apply it to a fresh checkout for evaluation. DeepSWE makes the agent commit its work and then extracts those commits through a [[verifier.collect]] hook declared in each task.toml, with grading in a separate verifier environment rather than a shared one. Committing is part of the protocol, not an optional step.

That design has a cost. An agent that solves the task but never commits produces nothing to grade, so the harness has to be told to commit, and a silent failure to commit looks identical to a wrong answer unless you read run.log. The benefit is a cleaner boundary: what gets graded is exactly what the agent chose to record as its work, extracted by a declared hook rather than by diffing a dirty tree. The other difference is scope. DeepSWE covers TypeScript, Go, Python, JavaScript and Rust, and it ships a language field in each task.toml, so you can slice results by language rather than reporting one aggregate number over a corpus that mixes them.

Maintenance, licensing and what an upgrade costs you

The repository is not archived, and the last push was on 2026-08-26. The README refers to a v1.1 grading change, and no releases were retrieved, so version tracking happens through the repository and the documentation rather than through tagged artifacts. The practical consequence is that a grading change can land without a version number you can pin against, which is why the Pier version floor matters more here than it would in a project with a release cadence.

The code is Apache-2.0. That is a permissive licence, and it covers the repository contents, including the tasks and the reference solutions in solution/. It does not tell you anything about the terms of the models you point at the benchmark, the API keys you export, or the Modal account you use for parallel sandboxes; those are separate agreements. The README does not state how task contributions are licensed or whether the held-out tests are covered by the same terms, and this is not legal advice. If you plan to redistribute the tasks or publish derived results commercially, read LICENSE and PROVENANCE.md at the repository root rather than assuming the top-level identifier settles it. PROVENANCE.md exists precisely because the tasks come from other repositories, so the origin of each task is a question that file is meant to answer.

Editorial conclusion

Adopt DeepSWE if you are comparing coding agents on multi-file, long-horizon work and you want the grading to happen outside the agent's own container. Skip it if you need a fast smoke test of a model's single-function ability, or if you cannot run Docker-backed sandboxes, since every task ships an environment Dockerfile and grading happens in a pristine container. Before you report a number, verify three things: that your Pier version is newer than 0.3.0, since v1.1 grading requires the separate verifier environment; that you are quoting reward.json rather than a pass fraction you computed yourself; and that the tasks you sampled came from --sample-seed, because the subset is deterministic and a hand-picked subset is not the benchmark.

Frequently asked questions

Official sources

1. datacurve-ai/deep-swe on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README