ProgramBench: asking agents to rebuild a program from its binary
Can Language Models Rebuild Programs From Scratch?
At a glance
- What is it?
- ProgramBench is a benchmark from Meta's research organization with a single, sharp question in its title: can language models rebuild programs from scratch? Given only a compiled binary and its documentation, an agent must architect and implement a complete codebase reproducing the original behaviour, with tests, leaderboard and a baseline published openly.
- Who is it for?
- ProgramBench fits agent developers and evaluation researchers who need a construction-oriented benchmark: rebuild a program's codebase from its compiled binary and documentation, scored on behaviour, with tests, leaderboard and a mini-swe-agent baseline all published under MIT.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 11 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 17, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The task: from binary back to codebase
Most coding benchmarks hand the model a repository and a bug report. ProgramBench inverts the direction of software engineering: the agent receives only a compiled binary and its documentation, and must architect and implement a complete codebase that reproduces the original program's behaviour. Nothing about the source is given, because the source is the deliverable.
That framing tests a different capability than patch generation. Rebuilding requires reverse-engineering behaviour from a running artifact, making architectural decisions the original authors made, and implementing them well enough to pass behavioural comparison. It is closer to greenfield engineering judged by fidelity than to repair work judged by diff.
The paper, arXiv:2605.03546, carries the title as a question, Can Language Models Rebuild Programs From Scratch?, and the author list reads like a roster of agent and software-engineering research. The benchmark's own framing as an open question rather than a leaderboard victory lap is the correct posture for a measure this young.
What is published, and where
The release is complete by research-benchmark standards. A website hosts the leaderboard and accepts submissions. The evaluation tests ship as a Hugging Face dataset, so the scoring material is public rather than hidden behind an API. A separate submissions repository collects entries, which makes the leaderboard's provenance auditable. A usage guide and a documented baseline complete the set.
The baseline is notable in itself: it runs through mini-swe-agent, an established minimal agent harness, rather than a bespoke runner. Publishing your baseline on someone else's maintained agent framework is a small decision that saves every adopter from reimplementing your glue.
The licence is MIT, the repository lives under the Facebook research organization, and the last push was on 2026-09-08, weeks before this review.
Running it, in three sizes
The harness installs the modern way, three ways:
uvx programbench --help
uv pip install programbench
pip install programbenchFor development on the benchmark itself:
git clone https://github.com/facebookresearch/programbench.git
cd programbench
uv sync # installs editable + dev dependenciesAnd the documented baseline, again two ways:
uvx --from mini-swe-agent mini-extra programbench --help
pip install mini-swe-agent && mini-extra programbench --helpThe pattern is consistent: the tool is a package first and a repository second, runnable without cloning anything, with the clone reserved for people changing the benchmark rather than using it. The usage guide the README points to covers the details this front page deliberately omits.
One reading of that structure: the project expects its audience to be developers wiring evaluation into agent pipelines, for whom a command-line entry point is the interface, and treats the repository as the artefact of record rather than the distribution mechanism.
Why this benchmark earns its slot
The field's dominant benchmarks measure repair: find the bug in a real repository, produce a patch, pass the tests. Those measure precision inside an existing architecture. ProgramBench measures construction: produce the architecture, then the code, from behaviour alone. Those are different claims about a model, and a leaderboard that only contains the first kind will systematically overanswer the second question.
There is also a falsifiability property worth naming. Behavioural equivalence against a compiled original is a harder standard to game than test-suite matching against tests written alongside the source, because the reference implementation exists as an artifact rather than as expectations. A rebuilt program that passes has reproduced something, not merely satisfied its authors' assumptions.
What the benchmark does not yet have, at this age, is a body of independent results. The leaderboard is the project's own, the baseline is the project's own, and the interesting question, where the frontier actually stands on reconstruction, is answered in the paper rather than by third parties. Adopters should read it as an instrument, freshly calibrated, not as consensus.
Who should use it, and against what
Two audiences. Agent developers who want a construction-oriented evaluation alongside their repair-oriented numbers, since a system that patches well and architects poorly is common and invisible on SWE-bench-style suites. And evaluation researchers studying what behavioural fidelity means for code generation, for whom the published tests and submissions make the benchmark a workable substrate.
The comparison set is well defined by the field. SWE-bench and its variants remain the reference for repository repair against real issues, and nothing here replaces them. Competitive-programming suites measure algorithmic generation in small, self-contained problems, the opposite scale of a whole program. ProgramBench's slot is the whole-program reconstruction middle, defined by the binary-to-codebase direction.
The practical costs are the usual ones for serious evaluation: agent runs against many programs, compute for the agents under test, and care in reading behavioural scores. The licence invites fork-and-extend, which for a benchmark is the right default.
Editorial conclusion
ProgramBench fits agent developers and evaluation researchers who need a construction-oriented benchmark: rebuild a program's codebase from its compiled binary and documentation, scored on behaviour, with tests, leaderboard and a mini-swe-agent baseline all published under MIT. It does not fit anyone wanting a quick single-number model ranking, since the benchmark is young and independent results are thin, or anyone measuring repair work specifically, which the SWE-bench family already serves. Verify first: the paper's methodology for what behavioural reproduction means, the leaderboard's submission rules, and whether your agent harness integrates the CLI cleanly. The last push was on 2026-09-08.
Frequently asked questions
What is a ProgramBench?
A benchmark from Meta's research organization testing whether language models can rebuild programs from scratch: given only a compiled binary and its documentation, an agent must design and implement a complete codebase reproducing the original behaviour, scored against published tests on a public leaderboard.
Can language models rebuild programs from scratch?
That is ProgramBench's titular research question. The benchmark measures it directly by scoring rebuilt codebases against the original programs' behaviour, and the accompanying paper reports where current models stand on that measure.
How do I run the ProgramBench baseline?
Through mini-swe-agent: either uvx --from mini-swe-agent mini-extra programbench, or pip install mini-swe-agent followed by mini-extra programbench. The benchmark harness itself installs with pip, or runs directly with uvx programbench.
Community notes