Terminal-Bench-Science: A 70-Task Benchmark for AI Agents on Scientific Research Workflows

What Terminal-Bench-Science Measures That a Coding Benchmark Does Not

Most agent benchmarks ask whether a model can patch a function or close a GitHub issue. Terminal-Bench-Science asks whether an agent can carry out a research workflow in a terminal, and the README states that tasks are authored and reviewed by domain experts across the life, physical, earth, mathematical, and engineering sciences. The unit of evaluation is therefore a workflow, not a diff. The repository currently documents 70 expert-curated tasks and says it is growing toward 100+. That count is the whole published scope; there is no per-domain breakdown in the material beyond a generated coverage graphic referenced as assets/task-coverage.svg. The audience is narrow and specific. On one side, evaluation teams who need a harder, more domain-grounded signal than software engineering suites provide. On the other, the scientists themselves: the README frames the project as a hub for domain experts to contribute tasks reflecting the work they want AI systems to support, with a proposal form, a contribution guide, and a review pipeline behind it. If you are looking for an installable package that adds capability to an agent, this is the wrong repository. It ships tasks and QA automation, and the agent is something you bring.

The Harbor Harness and the Oracle-First Data Flow

The mechanism is a harness plus a dataset, and the ordering matters. You install Harbor, then run the dataset's oracle solutions before touching any model. The README's own command runs the oracle five times with -k 5, which means each task is attempted five times and the pass pattern, not a single run, is the signal. Only after that does an agent trial make sense, because a task whose reference solution fails intermittently in your environment cannot distinguish a weak agent from a broken container. Task authoring mirrors this. The contribution flow is Propose, Build, Review: a proposal form for feedback and approval, then a pull request following CONTRIBUTING.md, then automated checks plus parallel domain and technical review and a final bar-raiser approval before merge. The commented-out section of the README describes the CI side in more detail than the visible text: static checks covering path validation, Dockerfile sanity, a canary, metadata and test references; an implementation rubric review run through a tool called harbor check against a 39-criteria rubric stored at rubrics/task-implementation.toml; TF-IDF similarity detection for duplicates; Docker build plus oracle and no-op validation, where the solution must pass and a no-op must fail; multi-agent trials; and adversarial cheat trials aimed at reward hacking. That list is the most interesting thing in the repository. A no-op-must-fail gate and a separate cheat-trial stage are the two checks most benchmarks skip, and they are the two that catch tasks solvable by reading the test file.

Install and Run Commands, and the Environment Assumption Behind Them

There are two install paths and one required runtime choice. Install Harbor with uv tool install "harbor[modal,daytona]" or pip install "harbor[modal,daytona]". The extras are not optional in practice: the README recommends Modal or Daytona for running the benchmark, and both example commands pass --env modal. The oracle command is harbor run -d terminal-bench-science/terminal-bench-science@latest -k 5 --agent oracle --n-concurrent 32 --env modal. The agent command swaps in --agent claude-code --model anthropic/claude-opus-5 and adds --ak reasoning_effort=max. Reading the flags as a set: -d names the dataset and its version tag, which is why the README points at tagged releases on Harbor Hub rather than a git ref; -k sets repetitions; --n-concurrent 32 sets parallel task execution, which is what makes the Modal or Daytona dependency load-bearing rather than cosmetic, since 32 concurrent containerized scientific environments is not a laptop workload; --agent selects the harness-side agent adapter; --model and --ak pass the model identifier and agent-specific keyword arguments through. The --ak flag is the one to watch. It is a passthrough, so its accepted keys depend on the agent adapter, not on the benchmark, and reasoning_effort=max is an example rather than a documented contract. The README also notes that running the oracle on your own setup may flake, and asks you to open an issue rather than patch around it. Treat that as a signal about the maturity of the container definitions across environments.

Where the Benchmark Breaks Down

The first limitation is stated by the project itself: the oracle can flake depending on the sandboxing environment. That is a reproducibility boundary, and it is the reason the README insists on the 5x oracle run before any agent measurement. If your environment is not Modal or Daytona, you are on a path the documentation does not exercise. The second is scope. Seventy tasks across five broad domains works out to roughly fourteen per domain, and the material gives no per-domain count, no difficulty distribution, and no statement about which tasks are considered saturated by current frontier agents. A benchmark that is described as continuous and evolving alongside frontier AI has a moving target problem: the README says it is growing toward 100+, which means scores are not comparable across dataset versions unless you pin one, and the -d flag with @latest does the opposite of pinning. The third is the verification constraint on contributions. Tasks must produce outcomes that can be objectively verified in a terminal environment. That requirement is what makes the benchmark automatable, and it is also what excludes a large share of real research work: anything judged by reading a figure, interpreting a spectrum by eye, or accepting a defensible-but-different analysis path cannot be scored this way. The benchmark measures the terminal-verifiable slice of science, which is a real slice but not the whole one. The fourth is that cheat trials and rubric review are described in a commented-out README block. The files they reference (TASK_REVIEW_AUTOMATION.md, rubrics/task-implementation.toml) are named there, but a reader of the rendered README will not see them, and the material does not confirm their current contents.

How It Differs From SWE-bench and From Running Your Own Eval

The closest familiar comparison is SWE-bench, and the difference is in what the task is anchored to. SWE-bench tasks are anchored to real repository issues and their merged patches, so the ground truth is a diff that human maintainers accepted. Terminal-Bench-Science tasks are anchored to expert-authored workflows with a solution script and a verifier, and the ground truth is the author's own reference implementation passing while a no-op fails. That shift buys domain breadth (life, physical, earth, mathematical, engineering) at the cost of the external validation a merged upstream patch provides. The second alternative is building your own evaluation harness around your team's real research scripts. That gives you tasks nobody else has and a signal directly tied to your work, but you inherit the entire QA problem: writing verifiers, catching reward hacks, detecting near-duplicate tasks, and confirming the reference solution is stable. The repository's CI pipeline, as described in the commented README block, is essentially a packaged answer to that problem, and it is the part most worth reading even if you never run the benchmark. The third alternative is simply running an agent on your own terminal tasks and reading the transcripts. That is cheaper and often more informative for a single team, but it does not produce a number you can compare against anyone else's.

Maintenance, Versioning and the Apache-2.0 Terms

The licence is Apache-2.0, which permits commercial use, modification and redistribution provided you keep the licence and notice files and observe the patent grant and termination terms. That is a permissive baseline and it says nothing about the tasks themselves: the material does not state whether individual contributed tasks carry separate terms, and it does not address whether task content is derived from published datasets or papers with their own conditions. If you plan to redistribute the dataset or fold it into a product, that is the question to resolve before shipping, and it is not answered in the README. On maintenance, the versioning is the cost centre. v0.1.0 landed on 2026-08-26 and the last push to main was 2026-09-09, so the project is young and active, and the README describes it as continuous rather than frozen. Pinning is therefore a deliberate act: -d terminal-bench-science/terminal-bench-science@latest tracks a moving dataset, and the README points to tagged releases on Harbor Hub for the alternative. Upgrading means re-running the oracle 5x against the new tag before trusting any score delta, because a change in the task set and a change in agent capability look identical in a single number. Budget for that re-run as a recurring cost, not a one-off.

A Concrete Reading of the Contribution Pipeline

If you are a domain scientist rather than an evaluation engineer, the part of this repository aimed at you is the pipeline, and it is worth being precise about what it asks. You propose through an Airtable form and wait for feedback and approval before writing anything. Then you build the task and open a pull request against CONTRIBUTING.md, which the README describes as a step-by-step guide. Then the task faces automated static checks, parallel domain and technical review, and a final bar-raiser approval. Two of those gates are the ones that will send work back. The oracle and no-op validation means your reference solution has to pass reliably inside a container and a do-nothing agent has to fail, so a task where partial credit is the natural outcome needs a verifier that still separates the two cases cleanly. The cheat trials mean a task whose answer is recoverable from the test file, the environment variables, or a fixture path will be caught before merge. Neither gate is unusual in principle. What is unusual is that they are described as automatic and applied to every PR rather than reserved for a curated subset. The practical consequence for a contributor is that the verifier design is the hard part of the task, not the scientific content, and the review cycle will surface that.

Editorial conclusion

Adopt it if you need a domain-grounded evaluation of a terminal agent and you can run Modal or Daytona, because the oracle must pass five times before any agent score means anything. Do not adopt it if you need a fast, dependency-light eval or a score that maps to a single scientific skill: the tasks span five domains and the repository does not publish per-task difficulty calibration. Verify first that the oracle run passes 5x in your own sandboxing environment, since the README explicitly asks you to report flakes rather than work around them.

Official sources

1. harbor-framework/terminal-bench-science on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases