SWE-bench Pro: Inside Scale's Long-Horizon Coding Benchmark and Its OS Repository

What SWE-bench Pro measures and who it is aimed at

The benchmark gives a model a codebase and an issue, and asks for a patch that resolves the described problem. That is the same shape as SWE-bench, which the README names as the inspiration. The difference the project advertises is horizon: the paper title asks whether agents can solve long-horizon software engineering tasks, and the repository is built around that question rather than around short, single-file fixes. The intended user is someone evaluating an agent scaffold, not someone looking for a library to drop into a product. You need a patch-generating harness, a way to collect its output, and a machine that can run containers. The README points at SWE-agent as the reference scaffold and at mini-swe-agent as an alternative, noting that mini-swe-agent results are comparable to SWE-agent for Sonnet 4.5. That comparison is the project's own claim, not an independent measurement, and it is scoped to one model.

The three-stage pipeline: generate, gather, evaluate

The workflow is deliberately split. Stage one is patch generation with your harness of choice; the README says the output will be .pred files containing model-generated patches for each instance. Stage two is collection: helper_code/gather_patches.py walks a directory of instance folders and emits a single JSON file in the format the evaluator expects, with each entry carrying instance_id, patch and prefix. Stage three is swe_bench_pro_eval.py, which takes the raw sample CSV, the patch JSON, an output directory, a scripts directory and a worker count. The separation matters because it lets you swap scaffolds without touching the evaluator, and it lets you re-score an existing patch set. It also means a failure in any stage produces a silent gap rather than an error in the others: if gather_patches.py misses a directory, the evaluator simply sees fewer instances. The evaluator runs the containers, so the patch JSON is the only contract between generation and scoring.

Docker images and the dockerhub_tag contract

Each instance in the HuggingFace dataset carries a dockerhub_tag column, and the README shows the mapping explicitly: full_image = f"jefzda/sweap-images:{docker_tag}". Prebuilt images live under the jefzda/sweap-images repository on Docker Hub. The evaluator takes --dockerhub_username=jefzda for the same reason. One warning in the README is easy to miss and expensive to ignore: bash runs by default in these images, so you should not manually invoke bash when running them. That is a container entrypoint convention, not a stylistic preference, and getting it wrong will change what your harness sees. The images are the main reason this benchmark is reproducible at all. Building per-instance environments from scratch across a long-horizon dataset is the part most teams underestimate, and the README explicitly recommends using the DockerHub images rather than building them from scratch. The trade-off is that you inherit whatever state those images are in, and the repository does not document an image versioning or rebuild policy.

Getting it running: dependencies, Modal and the local Docker beta

Installation is three steps. First, pip install -r requirements.txt. Second, install Docker, with the README linking the official install guide and, for Linux, the post-installation steps. Third, configure Modal, which the README marks as recommended: run modal setup, follow the prompts, then confirm the credentials in ~/.modal.toml, which should contain token_id, token_secret and active = true. Local Docker is listed as beta and needs no additional setup; you pass --use_local_docker when running evaluations. The evaluation command in the README is python swe_bench_pro_eval.py with --raw_sample_path=swe_bench_pro_full.csv, --patch_path, --output_dir, --scripts_dir=run_scripts, --num_workers=100 and --dockerhub_username=jefzda. The gather step is python helper_code/gather_patches.py with --directory, --prefix and --output. The default of 100 workers is a strong signal about the intended execution environment: on a single workstation, that number will not be reachable without heavy resource contention. The README also notes that gold patches are available in the HuggingFace dataset and that a helper script in helper_code extracts them into the required JSON format, which gives you a way to validate the evaluation path before spending money on model calls.

Where the material is thin or the design costs you

The README does not state per-instance runtime, image sizes, total dataset size or expected cost, so capacity planning is guesswork until you run a subset. The 01/07 news item says an issue with tutao instances where they take a long time to eval was fixed and the relevant run scripts updated, which confirms that evaluation duration has been a real problem for at least one instance family. The 2/9 item says some unit tests were removed because they were outdated, with the example that they required the year 2025, or were previously not intended to be included. That is a reminder that the grading criteria have changed after publication, so results from an older snapshot of the dataset are not directly comparable to current ones. The 05/18 item states that issues with the leaderboard have been identified and are being addressed, without saying which issues. If your goal is to compare against the public leaderboard, that open item should shape your timing. Finally, the SWE-agent integration is a git submodule, and the README tells you to see the official git documentation for submodule handling. That is an extra cloning step that trips up people who download a tarball instead of cloning with --recurse-submodules.

How this differs from running SWE-bench directly

SWE-bench is the reference point the README itself names, and the practical difference is in what the harness has to survive. SWE-bench Pro frames the task as long-horizon, which the README ties to the paper's central question, and the tooling reflects that: Modal-backed parallel evaluation, per-instance Docker images, and a scaffold submodule with configuration for model parameters and turn limits. If you already run SWE-bench, the migration cost is mostly in the plumbing rather than in the concept. You will need a Modal account or a local Docker setup, the jefzda image namespace, and a gather step that SWE-bench users may not have in their pipeline. The benefit is that the environment is prebuilt and the evaluator is a single script with a documented argument set. The cost is that you are further from the metal: when an instance fails, the first thing to check is whether the image and the dockerhub_tag from the dataset row still line up.

Reproducing leaderboard results and the maintenance bill

The README gives a four-step reproduction path: complete the SWE-agent submodule setup, run the scaffold (an example for Claude Sonnet 4.5 is included as claude.yaml, and vllm is supported for local models), compile predictions with gather_patches.py, then run swe_bench_pro_eval.py. The 10/28 news items say the SWE-agent scaffold reproduces the Sonnet 4.5 results and that mini-swe-agent was added with comparable results for the same model. The 10/3 item points to updated results without a cap limit at scaleapi.github.io/SWE-bench_Pro-os. Maintenance cost is real but hard to quantify from the README alone. You are tracking a moving dataset, a submodule with its own release cadence, Docker images under a third-party account name, and a leaderboard that the maintainers say is under repair. The licence is MIT, which is permissive for the code in this repository. Note that the dataset and the Docker images are separate artifacts with their own terms, and the README does not state what those terms are, so check the HuggingFace dataset card and the Docker Hub repository before redistributing either. Nothing here is legal advice.

Editorial conclusion

Adopt SWE-bench Pro if you need a long-horizon agent benchmark with prebuilt Docker images and a documented evaluation script, and you can afford Modal or local Docker plus the disk and orchestration that come with it. Do not adopt it if you want a single-command evaluation with no Docker dependency or if you cannot absorb the cost of running a scaffold plus containers per instance. Verify first that the DockerHub images you plan to use are current, since the 05/18 news item says leaderboard issues are being addressed and the images are hosted under the jefzda account rather than the project's own namespace.

Official sources

1. Issues
2. License: MIT
3. README
4. scaleapi/SWE-bench_Pro-os on GitHub