AgentEvals judges the path an agent took, not only the answer

Why the trajectory and not just the output

The premise is stated at the top of the documentation and it is the whole argument for the package.

Agentic applications give an LLM freedom over control flow in order to solve problems. While this freedom can be extremely powerful, the black box nature of LLMs can make it difficult to understand how changes in one part of your agent will affect others downstream. This makes evaluating your agents especially important.

The consequence of that freedom is that output-only evaluation is not enough. An agent can produce the right final answer after taking a path that calls the wrong tool, retries three times, and burns a hundred dollars of tokens. Nothing in the final string tells you that. The trajectory, the intermediate steps the agent takes as it runs, is where the behaviour lives, and that is what these evaluators are pointed at.

The package contains a collection of evaluators and utilities for evaluating agent performance with a focus on agent trajectory, and it is intended to provide a good conceptual starting point for your agent's evals. That last phrase is the right expectation to hold: these are reference implementations of common evaluation strategies, not a general evaluation framework.

The scope is deliberately narrow, and the documentation says where the wider tools are. If you are looking for more general evaluation tools, the companion package is openevals.

The project is MIT licensed, Python and TypeScript, with the last push on 2026-07-14 and recent releases for both languages, the js line at 0.0.7 from 2026-03-03 and the py line at 0.0.9 from 2025-07-24. The repository is split into python/ and js/ directories, plus scripts/, static/ and a uv.lock.

Installing it in either language, and the judge model you need

The install is a single package in either language.

For Python:

pip install agentevals

And for TypeScript:

npm install agentevals @langchain/core

The @langchain/core dependency in the TypeScript install is worth noticing, since it tells you the package is not standalone.

If you plan to use the LLM-as-judge evaluators, you also need an LLM client. By default, agentevals will use LangChain chat model integrations and comes with langchain_openai installed by default. However, if you prefer, you may use the OpenAI client directly, and then the install is `pip install openai` on the Python side or `npm install openai` on the TypeScript side.

That default is a real design decision. Using the LangChain chat model integrations means the judge can be pointed at any provider LangChain supports, not just OpenAI, so the same evaluator works with a local model or a different vendor. Going direct to the OpenAI client is the escape hatch when you do not want the LangChain dependency in the way.

The quickstart in the documentation uses an evaluator powered by OpenAI's o3-mini model to judge your results, so you will need to set your OpenAI API key as an environment variable:

export OPENAI_API_KEY="your_openai_api_key"

The documentation also notes it is helpful to be familiar with some evaluation concepts and with LangSmith's pytest integration for running evals, which is how the package expects to be driven in a test suite.

The first evaluator judges a fake weather trajectory

The example is worth reading in full, because it shows the shape of the data model and the shape of the result.

The trajectory is represented as a list of OpenAI-style messages, and the Python quickstart looks like this:

from agentevals.trajectory.llm import create_trajectory_llm_as_judge, TRAJECTORY_ACCURACY_PROMPT

trajectory_evaluator = create_trajectory_llm_as_judge(
    prompt=TRAJECTORY_ACCURACY_PROMPT,
    model="openai:o3-mini",
)

The outputs in the example are a fake trajectory, and the comment says so explicitly: in reality you would run your agent to get a real trajectory. The trajectory itself is a user message asking about the weather in SF, an assistant message with an empty content and a tool call to get_weather with the city argument, a tool message returning the temperature, and a final assistant message with the answer.

The result is printed as a dict with three keys: a key of trajectory_accuracy, a reasoning string, and a score of true. The reasoning is the interesting part, because it is the judge explaining the trajectory it saw, noting that the assistant recognises the goal, then efficiently makes a tool call, then communicates the result clearly, and therefore the score should be true.

That reasoning field is what makes the evaluator debuggable. When a trajectory fails, you get the judge's account of why, which is a starting point for a human rather than a bare false.

The TypeScript version mirrors this exactly with createTrajectoryLLMAsJudge and the same prompt constant, and the outputs are typed with a satisfies clause against FlexibleChatCompletionMessage[], so a malformed trajectory is a type error rather than a runtime surprise.

Four matching modes, and they answer different questions

The agent trajectory match evaluators are used to judge the trajectory of an agent's execution either against an expected trajectory or using an LLM.

The input format is specified precisely, which removes a common source of friction. These evaluators expect you to format your agent's trajectory as a list of OpenAI format dicts or as a list of LangChain BaseMessage classes, and they handle message formatting under the hood.

The API is create_trajectory_match_evaluator, createTrajectoryMatchEvaluator and create_async_trajectory_match_evaluator, and there are two customisation axes.

The first is trajectory_match_mode, or trajectoryMatchMode in TypeScript, which takes strict, unordered, subset or superset and provides the general strategy the evaluator will use to compare trajectories.

That list is the design decision in the package. A strict match demands the exact same sequence of steps, which is right when your agent's path is deterministic and you want any deviation to fail the test. An unordered match ignores the sequence and cares about the set of steps, which is right when your agent may take the same actions in a different order and you do not want to write a brittle test. Subset and superset are the asymmetric versions: subset means the actual trajectory must contain the expected steps, superset means it may contain more, and the pair together cover the common case of a reference that pins the critical calls and ignores the extras.

The second axis is tool_args_match_mode and tool_args_match_overrides, which customise how the evaluator considers equality between tool calls in the actual trajectory versus the reference. By default only tool c... and the documentation is cut off mid-sentence there, which is itself a reminder that the matching semantics for arguments are the part you will need to read in full before trusting a comparison.

Choosing the mode is the practical decision, and it depends on how stable your agent's path is. A strict match on an agent that reorders its tool calls will fail constantly and train you to ignore failures.

Graph trajectories extend the idea to LangGraph runs

The table of contents lists a separate set of evaluators for graphs, and the naming makes the relationship clear: agent trajectory match, trajectory LLM-as-judge, graph trajectory, graph trajectory LLM-as-judge and graph trajectory strict match.

The distinction is that an agent trajectory is a linear list of messages, while a graph trajectory is the path through a state graph with branches and conditional edges. An agent that takes different routes depending on what it finds needs an evaluator that understands the graph shape, not just a list.

The graph evaluators come in both flavours that the agent ones have, a strict match against a reference path and an LLM-as-judge, which suggests the same two approaches apply: assert the path when it is deterministic, and ask a model when it is not.

For a LangGraph user this is the part of the package that matters most, because a graph application is exactly the case where the intermediate steps carry the most information. A graph can loop, retry a node, or branch on a model decision, and the number of steps in the run is itself a signal worth asserting on.

The package also has a Python Async Support section, which matters for anyone with an async agent. An async agent that calls a tool at a time makes a linear trajectory check awkward, because the order of the calls is a property of the runtime rather than of the plan. The presence of create_async_trajectory_match_evaluator as a first-class API rather than an afterthought suggests the authors thought about that.

LangSmith is how the evaluators get run in a test suite

The last section of the documentation is about integration, and it is what makes the package usable rather than a set of functions.

There is a LangSmith Integration section with subsections for Pytest or Vitest/Jest and for Evaluate. The documentation points at LangSmith's pytest integration for running evals, and the pairing of the two languages is consistent with the rest of the package.

That integration is the point. An evaluator that returns a dict with a key, a score and a reasoning string is not much use on its own. Run under pytest, the same function becomes a test case: a trajectory regression is a failing test, with the judge's reasoning available as the failure output. That is the difference between an evaluation library and an evaluation workflow, and it is why the quickstart shows the result being printed rather than asserted.

The LangSmith Evaluate path is the other half, for running the same evaluators over a dataset of trajectories rather than as individual tests, which is what you want when you want a score for a release rather than a red build.

For an adopter the practical sequence is therefore: install the package, pick the evaluator that matches how stable your agent's path is, wire it into pytest or Vitest through LangSmith, and set the judge model as an environment variable if you are using the LLM-as-judge variants.

One caveat worth stating plainly. The LLM-as-judge evaluators spend a model call per trajectory and their verdict is a judgement rather than a proof, so a trajectory suite is a signal about behaviour, not a guarantee. The strict match evaluators are the ones that give you a hard pass or fail, and they are the ones to put in a blocking CI job if you want the suite to be authoritative.

Editorial conclusion

Adopt AgentEvals if you are building an agentic application on LangChain or LangGraph and need to catch regressions in how the agent gets to its answer, not just in the answer itself. Skip it if your application is a single call to a model rather than a multi-step agent, because trajectory evaluation has nothing to look at, and if you are not already on LangChain, since the package is built around LangChain message types and LangSmith. Verify first which matching mode matches how stable your agent's path is, because a strict match will fail on a reordering that a subset match would accept, and confirm that your judge model is one you are willing to spend on every evaluation run.

Frequently asked questions

Official sources

1. Issues
2. langchain-ai/agentevals on GitHub
3. License: MIT
4. README
5. Releases