# terminal-bench measures agents on hard terminal work, task by task

> Terminal-Bench is a benchmark of difficult, diverse tasks for coding agents that work in a shell, published as versioned dataset releases on Harbor Hub and run through the Harbor framework. The README's own advice is to run the oracle solutions five times on your own sandbox before you trust any agent score.

**harbor-framework/terminal-bench** — Measuring and evolving with the frontier of agent work

- Repository: https://github.com/harbor-framework/terminal-bench
- Website: https://tbench.ai
- Stars: 1,016 · Forks: 620
- Language: Python
- License: Apache-2.0
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/harbor-framework-terminal-bench

## A yardstick for agents that work in a shell

Terminal-Bench is a benchmark for agent work, and its subject is agents that operate in a terminal rather than answer in a chat window. The README describes it as a benchmark designed to measure the frontier of agent work with a diverse, difficult, high quality set of tasks that evolve over time, and it says that frontier agent builders use it to track progress and compare capabilities.

The audience is therefore narrow and specific. If you are building a coding agent, or choosing between models and agent harnesses, you need a task set that is hard enough to separate systems that differ and stable enough that you can compare a run today with a run next month. The homepage is tbench.ai, and the dataset and leaderboard live on Harbor Hub.

What it is not is a unit test suite or a general capability score. Nothing in the documentation suggests it measures chat quality, reasoning in the abstract, or anything outside the terminal. If your agent never opens a shell, this benchmark will not tell you much about it.

## The dataset is versioned on Harbor Hub, not pinned in the repository

The design decision that shapes everything else is that Terminal-Bench is a continuous benchmark. The README says so directly, and explains that tagged releases are published on the Harbor Hub rather than inside the repository. The repository holds the tasks directory, the leaderboard directory, a legacy directory, scripts and tools, but the version a run uses is fetched from the hub.

The identifier carries the version: terminal-bench/terminal-bench@latest is the moving target, and the release tags are the fixed points. The README points at tagged releases and at the dataset page for the latest published version.

That arrangement is good for benchmark integrity in one direction and awkward in another. A continuous dataset means new tasks arrive and difficulty drifts upward, so a score is only meaningful against the version it was taken on. The leaderboard link in the resources section pins a specific tag, 4-0-0, which shows the intended pattern: publish a run against a named version, then compare with other runs against that same version. Comparing a run on latest with a run on an older tag tells you about dataset drift, not about your agent.

## Install Harbor and run the oracle five times before anything else

The README's first instruction is not to test an agent. It is to check that the tasks work in your own sandboxing environment. Harbor is installed with uv, and the modal extra pulls in the Modal backend, which is what the README says the project uses for its own CI/CD and leaderboard experiments.

```bash
uv tool install 'harbor[modal]'
uv run harbor run -d terminal-bench/terminal-bench@latest \
   -k 5  \
   --agent oracle \
   --n-concurrent 500 \
   --env modal
```

The oracle agent is the built-in reference solution for the tasks, so running it tells you whether your environment can actually complete them. The README asks you to run it 5x, which is also the -k 5 flag in the command, so the repeat count and the key coincide. --n-concurrent 500 controls how many task instances run at once, and --env modal selects the execution backend.

The README is unusually candid about what to do when this fails. If the oracle flakes on your setup, the instruction is to open an issue. That is a meaningful detail for anyone planning to rely on the benchmark: the project treats environment differences as something to report rather than to hide.

## Testing an agent means passing an agent name and a model

Once the oracle is green in your environment, a real agent run is the same command with two extra flags. The README says it plainly: to test an agent and a model, pass --agent and --model.

```bash
uv run harbor run -d terminal-bench/terminal-bench@latest \
   --agent claude-code \
   --model anthropic/claude-fable-5 \
   --ak reasoning_effort=max \
   --n-concurrent 100 \
   --env modal
```

The agent name selects a harness, and the model identifier selects what that harness talks to. The --ak flag sets an agent keyword, here reasoning_effort=max, which is the lever for controlling how much the agent deliberates. Concurrency drops to 100 for the agent run compared with 500 for the oracle check.

One line in the README covers what happens when the agent itself misbehaves rather than the tasks: if your agent runs encounter any problems, please open an issue. That points at a maintained view of the failures agents produce, which is arguably as valuable as the scores themselves.

## Modal is the reference sandbox, so expect environment specific failures

The strongest warning in the documentation is about portability. The project develops its tasks using Modal in its CI/CD and its leaderboard experiments. Your sandbox is not Modal, and the README assumes you will find out what that means when the oracle flakes on your setup.

This is the main limitation to plan around. Terminal-Bench measures an agent, but it also measures the container, the filesystem and the tools the task assumes. A task that passes in the project's own environment and fails on yours is ambiguous, and the documented resolution path is a GitHub issue rather than a local configuration knob. The README does not document a list of supported sandbox environments, and the only backend named in the commands is modal, so an in-house sandbox would be uncharted territory.

The second limitation is the one built into the concept. A continuous dataset that evolves over time means the number you report is tied to a moment. It is a benchmark designed to be compared against a tagged release, not to be quoted as a single number that stays true.

## Task review is a pipeline with automated checks, not a merge button

The contributor path is unusually explicit for a benchmark, and it tells you how much curation sits behind the task set.

The README says the project is actively looking for contributors to add new, challenging tasks, and points at CONTRIBUTING.md as the technical guide for creating and submitting tasks. Before writing a full submission, it asks contributors to propose the task first and get feedback, using the section on proposing a task. There is also a task proposal rubric at docs/prompts/task-proposal.md, which is worth reading before you write anything, because it defines what a good proposal looks like to the reviewers.

Pull requests that add a task follow a review automation documented in docs/TASK_REVIEW_AUTOMATION.md. The described flow is that automated checks run on every push, a maintainer discusses feedback and triggers further checks, and the task is then iterated to a high quality bar. In other words, a task is not merged because it passes a script. A human still decides, and the rubric is the shared language for that decision.

For an evaluator, that process is a quality signal you can point at: the difficulty of the set is defended by an explicit review process rather than by whoever happened to submit.

## Version tags, Apache-2.0 and how the releases are moving

Terminal-Bench is released under Apache-2.0, with the licence file at the root of the repository. For academic use, the README asks you to cite the repository using the Cite this repository button on GitHub or the metadata in CITATION.cff, which is present at the top level.

The release history moves in large steps rather than small patches. The two most recent tagged releases are v4.0.0 from 2026-08-26 and v3.0.0 from 2026-07-23, and the last push to the repository was on 2026-10-22. Major version numbers mean the task set itself changes, which is exactly the behaviour you would expect from a dataset that is designed to keep getting harder.

The upgrade cost for a user is therefore a decision about versions rather than about code. The framework comes from Harbor, which is a separate install, and the dataset you evaluate against is a version string. If you want comparable numbers across a quarter, you pin the tag; if you want to know where the current frontier sits, you follow latest and accept that the two are different measurements.

## Conclusion

terminal-bench fits teams that ship agents and need one shared yardstick for shell work, and it costs one command to try because Harbor installs from PyPI with uv and runs the whole suite on Modal. It is the wrong tool for a single script or a one-off prompt, and it is not a stable leaderboard by design, since the dataset evolves continuously. Verify first that the oracle passes five times in your own sandbox, and record which tagged dataset version you used, because a score taken against terminal-bench/terminal-bench@latest is not comparable with one taken next week.

## FAQ

### How do I install and run terminal-bench?

Install the Harbor framework with uv tool install 'harbor[modal]', then run harbor run against the dataset on Harbor Hub. The README asks you to run the oracle solutions 5x first, with --agent oracle and --env modal, so you can confirm the tasks work in your own sandbox before testing an agent.

### How do I test my own agent and model with terminal-bench?

Pass --agent and --model to harbor run, as in the README example using --agent claude-code and --model anthropic/claude-fable-5. The --ak flag sets an agent keyword such as reasoning_effort=max, and --n-concurrent controls how many task instances run at once.

### What does it mean that terminal-bench is a continuous benchmark?

The dataset evolves over time, with tagged releases published on the Harbor Hub rather than frozen in the repository. That means a score is only comparable with other runs against the same tagged version, so record which version you used.

### What should I do if the oracle solution fails in my environment?

The README asks you to open an issue if the oracle flakes on your setup. The project develops its tasks using Modal in its CI/CD and leaderboard experiments, so environment differences are expected to be reported.

### How are new terminal-bench tasks reviewed?

Task pull requests follow the review automation described in docs/TASK_REVIEW_AUTOMATION.md, where automated checks run on every push and a maintainer discusses feedback and triggers further checks. Contributors are asked to read CONTRIBUTING.md and the task proposal rubric at docs/prompts/task-proposal.md, and to propose a task for feedback before writing a full submission.

## Sources

- [harbor-framework/terminal-bench on GitHub](https://github.com/harbor-framework/terminal-bench)
- [License: Apache-2.0](https://github.com/harbor-framework/terminal-bench/blob/main/LICENSE)
- [Project website](https://tbench.ai)
- [README](https://github.com/harbor-framework/terminal-bench/blob/main/README.md)
- [Releases](https://github.com/harbor-framework/terminal-bench/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/harbor-framework-terminal-bench