tau2-bench: A Simulation Harness for Customer Service Agents That Talk to Users and Call Tools

The evaluation gap tau2-bench was built to fill

Most agent benchmarks measure a single turn: give the model a task, check the output. Customer service does not work that way. The agent has to follow a written policy, decide when to call a tool, ask the user for missing information, and handle the user changing their mind mid-conversation. tau2-bench is a simulation framework for exactly that setting. Each domain bundles a policy the agent must follow, a set of tools the agent can call, a set of tasks, and optionally a set of user tools for the user simulator. The user side is simulated too, which is the part that separates this from a static question-answering benchmark. The target audience is agent developers who need to know whether their agent's behaviour holds up over a multi-turn exchange, not just whether the final answer string matches.

Five domains, two communication modes

The available domains are mock, airline, retail, telecom and banking_knowledge. The first four ship with the core install. banking_knowledge is behind the knowledge extra and adds a retrieval pipeline with configurable RAG, document search, embeddings and agentic shell-based search. On top of the domains sit two modes. Text mode is half-duplex: turn-based chat with tool use, which is what most people mean by an agent evaluation. Voice mode is full-duplex and audio native, running end-to-end through realtime providers (OpenAI, Gemini, xAI). The orchestrator README is the place to read about how those two modes differ, because full-duplex changes the loop: the agent and the user can speak at the same time rather than taking strict turns. That is a meaningfully harder setting than text, and the two modes should not be treated as interchangeable when you report numbers.

What a task actually contains

The evaluation doc is the one to read before you trust a score. It explains what evaluation_criteria.actions means, how reward_basis gates the reward, and how to inspect action correctness. That phrasing matters: the reward is not a single opaque number produced by a judge model. It is gated by a declared basis, and the expected actions are checked against what the agent did. In practice this means a task can fail because the agent called the wrong tool, called the right tool with wrong arguments, or skipped a step the policy required, even if the user was satisfied. The task fixes released with v1.0.0 are instructive here. More than 75 fixes across airline, retail and banking removed incorrect expected actions, clarified ambiguous instructions, fixed impossible constraints, and added missing fallback behaviours, based on analysis from SABER (Cuadron et al., 2025). A benchmark that needed fixes of that kind had tasks whose ground truth was debatable, and that is worth remembering when you compare your agent against a published number.

Getting it running: uv, .env, and one tau2 command

Installation uses uv rather than pip install -e ., which is a change from tau2-bench and is called out in the README. Clone the repository, then run uv sync for the core text-mode domains. Extras are opt-in: uv sync --extra voice, --extra knowledge, --extra gym for a gymnasium RL interface, --extra dev for pytest, ruff and pre-commit, or --all-extras. Python must be >=3.12 and <3.14, up from >=3.10 in the previous line. Voice features also need system dependencies, with brew install portaudio ffmpeg given as the macOS example. API keys go in a .env file copied from .env.example; the framework routes through LiteLLM, so any provider LiteLLM supports should work. A first evaluation looks like this: tau2 run --domain airline --agent-llm gpt-4.1 --user-llm gpt-4.1 --num-trials 1 --num-tasks 5. Results land in data/simulations/, and tau2 view browses them. tau2 intro prints an overview of domains, commands and examples, which is the cheapest way to orient yourself before reading the CLI reference.

The v1.0.1 grading break is not a footnote

In July 2026, v1.0.1 fixed a couple of banking_knowledge task errors. The README is explicit about the consequence: results produced with tau2-bench below 1.0.1 are not comparable with results at or above 1.0.1, and affected leaderboard submissions were re-graded. Two escape hatches exist. Old results files can be re-scored with tau2 evaluate-trajs --fresh-tasks, and the pre-v1.0.1 tag reproduces pre-fix behaviour. Other domains are unaffected. This is the kind of detail that decides whether a benchmark is usable for regression testing. If your team pins an older version and later upgrades, your historical scores for that domain are not a baseline. If you are comparing against a paper or a leaderboard entry, check the version before you conclude your agent is better or worse. The README also notes a backward compatibility split: if you are evaluating an agent rather than training one, use the base task split, which is the default and matches the original tau-bench structure.

Where tau2-bench is the wrong tool

This is a customer service simulator, and the domains are the ones shipped: airline, retail, telecom, banking_knowledge, mock. If your agent operates a code repository, a database console, or a browser, the policy-and-tools abstraction will not map cleanly onto your work without you writing a new domain. The framework is also not a cheap smoke test. Every run consumes two LLM streams, one for the agent and one for the user simulator, and --num-trials exists precisely because a single trial is noisy. Voice mode adds realtime audio APIs on top of that. And there is a real cost to the task fixes: a benchmark whose expected actions were partly wrong until v1.0.0 and partly wrong in one domain until v1.0.1 has a shorter track record than its version number suggests. If you need a benchmark with years of stable ground truth, this is not it yet.

How this differs from SWE-bench style evaluation

SWE-bench and its relatives grade a single artefact: a patch that either makes a test suite pass or does not. There is no user, no policy, and no turn-taking. tau2-bench grades a trajectory. The agent has to satisfy a policy, call tools correctly, and extract information from a simulated user who may not volunteer it. That difference shows up in the failure modes you will actually observe. In a patch-based benchmark, a wrong answer is usually a wrong answer. Here, an agent can produce a correct final outcome through a policy-violating path and still be penalised, because reward_basis gates what counts. It also means the user simulator is part of your measurement apparatus. A weaker user model can make the agent look better by asking clearer questions, and the README's example passes the same model to --agent-llm and --user-llm, which is a reasonable default but not a neutral one. If you change one, change both and say so.

Maintenance, upgrades and the MIT licence

The project is active: last push September 2026, with v1.0.1 in July 2026, v1.0.0 in March 2026, and v0.2.0 in October 2025. Upgrades are not free. Moving from tau2-bench to the current line changed the installer from pip install -e . to uv sync, raised the Python floor to >=3.12, and refactored some internal APIs, with CHANGELOG.md as the reference. The v1.0.0 release also restructured tasks, so a pinned commit is the only way to reproduce an older score exactly. The licence is MIT, which is permissive and places few obligations on how you use or redistribute the code, though the benchmark data and any model outputs you generate carry their own considerations. That is a description of the licence identifier, not legal advice; check with your own counsel if redistribution matters to you. Practically, the maintenance burden is not the code. It is re-baselining: every grading fix invalidates prior numbers for the affected domain, and the project has shipped two such changes in the last year.

Editorial conclusion

Adopt tau2-bench if you are building a customer service agent and want a repeatable score on policy adherence, tool calls and user interaction, since the domain, policy, tools and tasks are all files you can edit. Do not adopt it if you need a single leaderboard number to compare against published results without checking versions, because v1.0.1 changed banking_knowledge grading and the README states that results below 1.0.1 are not comparable with results at or above it. Before running anything, verify which task split you are on (base versus the newer splits), confirm your Python is >=3.12 and <3.14, and decide whether you need the voice or knowledge extras, since each pulls a different dependency set.

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. sierra-research/tau2-bench on GitHub