Terminal-Bench: A Task Suite and Sandbox Harness for Command-Line Agents
A benchmark for LLMs on complicated tasks in the terminal
At a glance
- What is it?
- Terminal-Bench packages roughly 100 terminal tasks with test scripts and oracle solutions, plus a harness that wires a model to a Docker sandbox. It is a beta benchmark, and its own README now points new users at a successor project called harbor.
- Who is it for?
- Adopt Terminal-Bench if you need a reproducible terminal-agent evaluation and can accept a beta task set of roughly 100 items with no retrieved releases. Do not adopt it if you need a stable, long-lived harness: the README itself directs new users to harbor for running Terminal-Bench 2.0.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 67 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The Gap Terminal-Bench Fills: Agents That Have to Touch a Real Filesystem
Most agent evaluations ask a model to produce text. Terminal-Bench asks it to change a machine. The README frames the scope as "compiling code to training models and setting up servers," and describes the goal as evaluating "real-world, end-to-end tasks" handled autonomously. That distinction matters because a terminal task has state. A package gets installed, a file gets edited, a process gets started, and the grading script inspects the result rather than the transcript. The intended audience is narrow and identifiable: people building LLM agents, people building benchmarking frameworks, and people who want to probe system-level reasoning under conditions they can rerun. If your evaluation only needs a single-turn answer, this is heavier machinery than the problem requires.
Two Halves: A Task Folder and a Sandbox Harness
The project splits cleanly into a dataset and an execution harness, and the README treats that split as the core architecture. Each task carries three artifacts: an instruction written in English, a test script that decides whether the agent finished the job, and a reference solution the README calls the "oracle" solution. Tasks live in the tasks folder of the repository. The harness is the other half. It connects a language model to a sandboxed terminal environment, and the README states the package depends on uv and Docker. That dependency list is the honest summary of the design: the sandbox is a container, so anything that cannot run inside a Docker container is outside the benchmark's reach. There is no separate scoring service described in the material; the test script shipped with each task is the grader.
Installing and Running the Harness
The README gives two install paths, and they differ in tooling rather than outcome. The first is uv tool install terminal-bench. The second is pip install terminal-bench. Both produce the tb command-line interface. Once installed, the README's own next step is to inspect the harness options with tb run --help, and it points to the documentation for the full set of flags. Leaderboard evaluation uses a named dataset rather than the default task set. The README shows a run that passes --agent terminus, --model anthropic/claude-3-7-latest, --dataset-name terminal-bench-core, --dataset-version 0.1.1, and --n-concurrent 8. Read that example carefully before copying it: as printed in the README, only the first line ends with a backslash continuation, so the remaining flags sit on their own lines. Treat the block as a list of the flags that matter, not as a shell script to paste. The dataset pair is the part worth remembering: terminal-bench-core at version 0.1.1 is the beta task set that corresponds to the current leaderboard.
Versioning Is the Whole Reproducibility Story
A benchmark number is only as useful as the task set behind it, and Terminal-Bench handles this with named datasets and explicit versions. The README states that Terminal-Bench-Core v0.1.1 is the set of tasks for the beta release and corresponds to the current leaderboard, and it directs readers to a registry overview for how datasets and versioning work. This is the correct instinct, and it also creates a maintenance obligation. Any run that omits --dataset-name and --dataset-version is not comparable to a leaderboard entry, because you cannot tell from the command alone which tasks were executed. If you publish results, record both flags. The material does not describe how task revisions are handled across versions, so treat cross-version comparisons as an open question rather than an assumption.
Beta Status, Task Count, and the Successor Project
Three facts constrain adoption more than any feature does. First, the README states the project is "currently in beta with ~100 tasks." That is a small suite, and the tilde is the project's own hedge. Second, the README carries an announcement that new users should check out harbor, described as a new framework that can be used to run Terminal-Bench 2.0. A repository whose own front page redirects newcomers is telling you something about where active work is going. Third, no recent releases were retrieved for this repository, so there is no published version history in the supplied material to reason about upgrade cadence. None of this makes the task format unusable. It does mean you should decide whether you are adopting the benchmark's task definitions or its harness, because those two decisions now have different answers.
Where Terminal-Bench Is the Wrong Tool
The Docker requirement is a hard boundary. If your agent's job involves hardware, a GUI, a network topology you cannot reproduce in a container, or a proprietary environment you cannot ship, the harness cannot host it. The oracle solution requirement is a second boundary, and it is easy to underestimate. Every contributed task needs a reference solution that passes its own test script. That is a real engineering cost per task and it pushes the suite toward problems that are cleanly solvable and cleanly checkable. A task whose correct outcome is ambiguous, or whose success depends on judgement rather than an exit code, will be awkward to encode. Finally, roughly 100 tasks is a small sample. Differences of a few points between two agents on this suite should not be read as a ranking without checking which tasks moved, and the material does not supply per-task variance data to help with that.
What You Would Use Instead
The natural alternative is not a different benchmark but a different layer of the same one. The README points to harbor as the framework for running Terminal-Bench 2.0, which suggests a split where harbor owns execution and Terminal-Bench owns tasks. If your goal is to evaluate an agent today, that is a meaningfully different proposition from installing terminal-bench and driving tb yourself: one path uses the harness shipped in this package, the other uses a separate framework against a later task set. The material also mentions adapters, with a linked guide on creating a new adapter for a new benchmark, which implies the intended extension pattern for people who want to bring their own tasks rather than adopt these. Choosing between these paths is a question about which component you actually need, and the README's announcement is the clearest signal available about which one is receiving attention.
Licence and Maintenance Cost
The repository is Apache-2.0, which permits commercial and modified use and includes an explicit patent grant. That is a permissive licence, and it means wrapping the harness inside an internal evaluation pipeline is not obviously blocked by licensing. This is not legal advice; read the licence text and your own obligations if you plan to redistribute modified task definitions. On maintenance, the material supports a few concrete observations and no more. There are no retrieved releases, so there is no changelog to plan upgrades around. The README invites contributions, especially new and challenging tasks, which means the task set can shift under you between runs. Pinning --dataset-version is the only defence the documentation offers against that drift. Budget for the Docker and uv prerequisites as part of setup cost, since both are named as dependencies rather than optional extras.
Editorial conclusion
Adopt Terminal-Bench if you need a reproducible terminal-agent evaluation and can accept a beta task set of roughly 100 items with no retrieved releases. Do not adopt it if you need a stable, long-lived harness: the README itself directs new users to harbor for running Terminal-Bench 2.0. Before committing, check the docs installation page for the exact uv and Docker prerequisites, and run tb run --help to see which flags your installed version exposes.
Community notes