# scenario tests agents by simulating the user, not the unit

> scenario is an agent testing framework where the test is a simulated conversation between your agent and a model-driven user simulator, with assertions and judges placed at any point in the script. It plugs into pytest or vitest and works through a single call() method on your side.

**langwatch/scenario** — Agentic testing for agentic codebases

- Repository: https://github.com/langwatch/scenario
- Website: https://scenario.langwatch.ai
- Stars: 1,020 · Forks: 93
- Language: TypeScript
- License: Apache-2.0
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/langwatch-scenario

## The unit of testing is a conversation, not a function call

Scenario describes itself as an agent testing framework based on simulations, and the list of what it can do starts with the one thing ordinary tests cannot do: test real agent behaviour by simulating users in different scenarios and edge cases. That is the whole premise. A request and response pair tells you whether the agent produced a particular string. A simulation tells you what the agent does when a user asks a second question, changes their mind, or never supplies the detail the agent needs. The framework is built around that idea, and the practical consequence is that your test file looks like a scenario description rather than a function under test. It is aimed at teams who already have an agent and have run out of conventional ways to check it, and it is agnostic by design about which LLM evaluation framework sits behind the assertions, so it can be combined with existing eval tooling or with custom evals.

## The script is the turn-by-turn control surface

The mechanism that makes a simulation repeatable is the script. Each step in the script says who acts next: scenario.user() lets the user simulator produce a message, scenario.agent() lets the agent respond, and a custom assertion can be dropped between two steps to inspect the state at that exact point. The framework advertises the ability to evaluate and judge at any point of the conversation, and the script is how you say where. A simulation ends when you say so, with scenario.succeed() marking a successful run, and the result object carries a success flag that the test framework asserts on. This control is the difference between a simulation and a chat transcript. Without it, a simulation would be a non-deterministic conversation you can only judge at the end, and the tool-call check that happens mid-conversation would be impossible to express.

## Your side of the integration is one call() method

The framework asks for very little from the code under test. Integrating an agent means implementing just one call() method, and the documented Python shape is a class extending scenario.AgentAdapter that sets a name and returns the agent's response for the given input. Everything else, the user simulator, the judge, the assertions and the run bookkeeping, comes from the framework. The message list is handed to your agent, and the return value is what the simulator reacts to, so any agent that can be expressed as a function over messages can be tested this way, whether it is a local model call, a hosted endpoint or a stub. The example wraps a litellm completion call and caches it with a scenario.cache() decorator, which matters when the same conversation is replayed during a debugging session, since each uncached step is a billable model call.

## Judge agents read the transcript against written criteria

Assertions cover the mechanical facts, and the framework adds a second kind of check for everything else. A judge agent is given a list of criteria and evaluates the conversation against them, and the documented example passes five separate criteria to scenario.JudgeAgent, from a limit on follow-up questions to a requirement that the recipe contain no meat. Each criterion is a sentence, and the judge decides whether the agent met it. That design keeps evaluation criteria in readable form rather than in code, and it makes them easy to change, which is the point when a product requirement is still moving. The framework also states that it can be combined with any LLM eval framework or with custom evals, so a team can keep its existing scoring and add scenario simulations alongside rather than replacing what it has.

## Installing and running the first scenario

The quick start installs the package and a test runner, and the choice follows the language of the project:

```bash
uv add langwatch-scenario pytest
```

```bash
pnpm install @langwatch/scenario vitest
```

In Python, the test file imports scenario, sets a default model with scenario.configure(default_model="openai/gpt-4.1-mini"), and marks the test with the pytest markers agent_test and asyncio. The run itself names the simulation, describes the situation the user is in, and lists the agents that will take part:

```python
def check_for_weather_tool_call(state: scenario.ScenarioState):
    assert state.has_tool_call("get_current_weather")

result = await scenario.run(
    name="checking the weather",
    description="""
        The user is planning a boat trip from Barcelona to Rome,
        and is wondering what the weather will be like.
    """,
    agents=[
        WeatherAgent(),
        scenario.UserSimulatorAgent(model="openai/gpt-4.1-mini"),
    ],
    script=[
        scenario.user(),
        scenario.agent(),
        check_for_weather_tool_call,
        scenario.succeed(),
    ],
)

assert result.success
```

The assertion reads the state for a tool call by name, so the check knows which tool it wants. The TypeScript version follows the same shape with the same script steps, and the repository keeps full examples under the python/examples and typescript/examples folders.

## The JavaScript package ships vitest integration as subpath exports

The root package manifest is a private workspace manifest rather than the published package, which is worth knowing before you read its version number. It is marked private and carries version 0.4.0, while the published JavaScript line is on a different number, v1.7.0 as of 2026-10-07. What the manifest does show is the shape of the JavaScript distribution: the main entry is javascript/dist/index.js, the module entry javascript/dist/index.mjs, and the types entry javascript/dist/index.d.ts, with CommonJS and ES module entry points exported side by side. Four additional subpath exports exist for the vitest integration, one each for a reporter, a setup module, a global setup module and a config module, so a project wires the framework into vitest through those paths rather than through a plugin API. The two runtime dependencies are the OpenTelemetry Node SDK and langwatch, which tells you the run is traced and that the framework is part of the Langwatch toolchain rather than a standalone library.

## The README mentions Go, the tree ships Python and TypeScript

There is a mismatch worth flagging before you plan an adoption. The feature list states that the framework is available in Python, TypeScript and Go, but the repository tree contains python/ and javascript/ directories, and the release history shows separate python and javascript version lines, with python v1.6.0 and javascript v1.7.0 published on 2026-10-07 and python v1.5.0 on 2026-09-06. The Go client is not in this tree, so if you need it you should confirm where it comes from and how it tracks the two languages that are here. A second mismatch is the version number in the root manifest, which does not correspond to a published release. Neither is fatal, and both are the kind of detail that turns into an afternoon of confusion if you assume the manifest and the release notes agree.

## Where the model bill and the flakiness come from

The honest limitation of the approach is that a simulation is not a deterministic function. The user simulator is a model, the judge is a model, and your agent is usually a model, so three non-deterministic pieces sit between a green test and a red one. A simulation that passes today can fail tomorrow because the simulated user phrased the request differently, which means the suite has to be treated as a behavioural signal rather than as a hard gate on every commit, and the caching decorator on the agent call exists for a reason. The framework also assumes a model provider is reachable from the test environment, since the documented setup configures a model by name. For teams without model access in CI, that assumption is disqualifying, and for teams that pay per call, the cost of a large simulation suite is a budget question, not just a technical one.

## Conclusion

scenario fits teams shipping LLM agents who need behaviour checks that a request and response pair cannot give them, especially checks on tool use and on how the agent handles a user who changes the subject mid-conversation. It does not fit a service whose agent is a plain HTTP endpoint with no model in the loop, and it is a poor fit for a test suite that must run without model access, since the user simulator is itself a model. Before adopting it, check three things: the call() method on your side, which is the only integration point the framework requires; which model the user simulator and the judge will use, since default_model is a global setting in the documented setup; and the version you pin, because the Python and JavaScript packages are released on separate lines, with python v1.6.0 and javascript v1.7.0 both published on 2026-10-07. The last push was on 2026-10-14, and the licence is Apache-2.0.

## FAQ

### How do I install scenario and run my first agent test?

Install the package with a test runner, using uv add langwatch-scenario pytest for Python or pnpm install @langwatch/scenario vitest for TypeScript. Then write a test that calls scenario.run with a name, a description, the agents and a script, and assert on result.success.

### What do I have to implement to test my agent with scenario?

One call() method. The documented Python shape extends scenario.AgentAdapter, sets a name, and returns the agent response for the given input, and the framework supplies the user simulator, the judge, the assertions and the run bookkeeping.

### How do I check that my agent called a specific tool?

Write an assertion that reads the scenario state, as in assert state.has_tool_call("get_current_weather"), and place it in the script between the agent step and the end of the run. The script is what makes the check happen at a chosen point in the conversation rather than only at the end.

### Can scenario judge the quality of an agent response?

Yes, through a judge agent. The documented example passes a list of written criteria to scenario.JudgeAgent, such as a limit on follow-up questions or a requirement that a recipe contain no meat, and the judge evaluates the conversation against them.

### Which languages and versions does scenario publish?

The feature list names Python, TypeScript and Go, while the repository tree holds python/ and javascript/ directories. The published releases are versioned per language, with python v1.6.0 and javascript v1.7.0 released on 2026-10-07 and python v1.5.0 on 2026-09-06.

## Sources

- [langwatch/scenario on GitHub](https://github.com/langwatch/scenario)
- [License: Apache-2.0](https://github.com/langwatch/scenario/blob/main/LICENSE)
- [Project website](https://scenario.langwatch.ai)
- [README](https://github.com/langwatch/scenario/blob/main/README.md)
- [Releases](https://github.com/langwatch/scenario/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/langwatch-scenario