# autoevals: model-graded scorers as editable YAML templates, and a gateway you inherit by default

> AutoEvals is a dual-language library from Braintrust that ships a catalogue of evaluation templates for subjective tasks, with the prompts kept in YAML files rather than in code so a single example can be run, edited and inspected. Two details decide how you should use it. The statistical scorers are an optional extra, and with no endpoint configured your evaluation traffic goes to a hosted gateway rather than to your model provider.

**braintrustdata/autoevals** — AutoEvals is a tool for quickly and easily evaluating AI model outputs using best practices.

- Repository: https://github.com/braintrustdata/autoevals
- Stars: 1,044 · Forks: 85
- Language: Python
- License: MIT
- Published: 2026-09-30 · Updated: 2026-09-30 · Language: en
- Canonical page: https://hysenlabs.com/projects/braintrustdata-autoevals

## The scorers are YAML templates on disk, not functions in a library

The design decision that separates this from a scoring library is in the file list rather than in the prose. There is a directory of templates, and the getting started section points at a specific file inside it for the factuality prompt. So the graders are prompt templates stored as data, and the library loads them, renders them and parses the response. That has three consequences that matter in practice. You can run a single example and print the result, which is the stated reason the project exists in its current form. You can edit a prompt and see what changes, without writing a test harness. And you can read the exact instruction the grader was given, which is the difference between trusting a score and debugging one. The readme says many of the evaluations are adapted from an existing open evals project but are implemented so you can run them on individual examples, tweak the prompts and debug their outputs, and that sentence is the whole product statement. Three families of scorer are bundled, and the readme gives an example of each: a model acting as a judge, a heuristic such as edit distance, and a statistical method such as a similarity score for text. There is also a separate document at the top level enumerating the scorers, so the catalogue of what you can call is one file rather than a set of import paths, and a directory of evaluations alongside the templates. The type of task the model-graded ones target is stated plainly, fact checking and safety among them, and those are the cases where the right answer is a judgement rather than a string comparison.

## Four Python dependencies, and each one is a feature

The Python manifest declares four runtime dependencies, which is few enough to read as a specification of what the library actually does. A templating engine for moustache-style syntax is one, which is how a prompt template with placeholders gets filled with your input, output and expected values. A schema validation library is two, which is how a template or an evaluation definition is checked before a model is asked to grade it. An edit distance library is three, which is the heuristic scorer. A YAML parser is four, which is how the templates are loaded at all. Every one of those four is a feature rather than an accident, and none of them is a framework. The JavaScript package mirrors the same design with its own choices, and the comparison is instructive. It bundles a JSON schema validator, an edit distance library, a YAML parser, a cosine similarity helper, and a linear sum assignment library, which is the algorithm behind optimal one-to-one matching and is the kind of thing a scorer needs when the question is whether two sets of items match up in the best possible way rather than greedily. It also declares a schema library as a peer dependency rather than a direct one, accepting either of two major versions, which is the right call for a library whose whole job is to validate your data with the schema instance you already have rather than shipping a second copy of a validator you might disagree with. So the two implementations are not translations of each other. They are the same idea expressed in each ecosystem's idioms, which means a scorer available in one language is not automatically available in the other, and the scorer reference is the place to check.

## People's Republic of China versus China, and the rationale field

The getting started example is a single question and two answers, and it is the best possible argument for model-graded evaluation. The question is which country has the highest population. The output is the full formal name of a country, and the expected value is the short name. An edit-distance scorer would call that a near miss and score it low, and it would be wrong, because the two strings name the same thing and the grader has no way of knowing that from the characters. That is the failure mode of every string-matching scorer on a task where the acceptable answer is any of many phrasings, and it is why the model-graded family exists. The result object is the second thing to notice. The evaluator returns a score on a scale from zero to one, and the metadata carries a rationale along with the raw outputs the evaluator itself produced. The score is what you would put in a chart, and the rationale is what you read at two in the morning when a number moved and nobody can explain it. That is the whole reason to prefer a grader that shows its work over one that returns a number, and it is also the thing that makes these evaluations debuggable. One inconsistency in the documentation is worth knowing about if you are copying the examples, because one of the sample snippets logs the whole result object rather than the score and the metadata separately as the others do. The synchronous and asynchronous entry points on the same evaluator also matter for anything running in a loop, since an evaluation suite is many requests and the async form is the difference between a sequential minute and a concurrent one.

## The endpoint resolution order ends at a hosted gateway

This is the part to read before you install anything, and it is a few sentences in the middle of the readme. Authentication defaults to an environment variable holding an OpenAI key. Then the base URL for requests is resolved: first an environment variable pointing at an OpenAI-compatible endpoint, and if that is not set, a second environment variable for a different gateway, and if that is also not set, the default is a gateway hosted by the vendor, linked from the readme. So the default path for a user who sets an API key and nothing else is not their model provider. It is a proxy operated by the company that maintains this library, and every prompt you grade and every completion you grade it against passes through it. The readme is clear that this is a feature rather than an accident, listing simplified access to many providers, reduced costs from automatic request caching, and increased observability if you also enable logging to the vendor's platform. Each of those is real, and the caching in particular can cut the cost of a large evaluation run substantially. But the decision is yours and it has to be made deliberately. Automatic caching also means a repeated request may be served from a cache rather than sent to a model, which is exactly what you want for a stable evaluation suite and not what you want while iterating on a prompt. A second environment variable authenticates the gateway, and a separate key variable appears in the environment example file alongside the model provider one. If your evaluation data cannot leave your network, set the base URL explicitly to something you control, or pass your own client, which the next section covers. A hosted endpoint that is free while in beta is free until it is not.

## Statistical scorers are an extra, and the all extra leaves out the multi-provider one

The readme lists three families of scorer as bundled, and the manifest tells you which of them you actually get from a plain install. The base install carries the templating engine, the schema validator, the edit distance library and the YAML parser. The numerical libraries are not there. They are declared as an optional extra that pulls in a numeric stack, which is what a statistical scorer such as the text similarity score needs, and if you install the base package and call one of those scorers you will find the import missing. The second optional extra is the one that gives you access to many model providers through a translation library, and it comes with a bounded version range, pinned between two releases, which is a sign that the integration is known to break outside that window and is maintained as such. Then there is an extra called all, and this is where a small thing becomes worth flagging:

```toml
scipy = ["numpy", "scipy"]
litellm = ["litellm>=1.60,<1.85"]
all = ["autoevals[scipy]"]
```

The all extra expands to the numerical extra and nothing else. It does not include the multi-provider extra, and it does not include the documentation extra. So a user who reads the name, installs everything, and then tries to grade with a non-OpenAI model provider gets an import error, and the fix is a second install with the right extra. It is a one-line manifest oversight rather than a design problem, but it is the kind of thing that costs twenty minutes, and the cost is higher in a container image where the extra has to be added to a build file. The same pattern appears in the JavaScript package, where the multi-provider support would have a comparable dependency and the manifest's dependency list is not available in full here. The practical advice is to decide which scorers you need before you install rather than after, and to read the scorer reference document rather than assuming the readme's three bullets are a complete inventory of what a default install can run.

## Two languages, two version lines, and a package manager guard for contributors

The release record explains how to think about versions here, and the answer is that there is no single number. The recent releases are tagged per language, one for the Python package and one for the JavaScript package, and the two most recent were published about half an hour apart on the same day at the same version. So the two ecosystems version independently, and a lock file pinning the JavaScript package tells you nothing about the Python one in the same repository. The two manifests also get their versions differently. The Python project declares its version dynamically, sourced from an attribute inside the package, which is the convention when a build tool derives the version from the code. The JavaScript manifest carries the version as a literal string. Both are fine; they just mean you read them in different places. The installation story is the interesting part, because the package is published for both package managers and yet the repository actively refuses one of them:

```json
  "engines": {
    "pnpm": ">=10.27.0",
    "npm": "please-use-pnpm",
    "yarn": "please-use-pnpm"
  },
```

Read that as a contributor guard rather than a consumer restriction, because the install hook behind it only fires when the install is happening in the repository itself, checking whether the current working directory is the initial one and whether the package manager identifies as the required one, and it exits with an error if not. As a consumer installing the published package from a directory of your own, the check does not trigger and either manager works. Inside the repository, a wrong package manager aborts the install. That is a deliberate choice, and it is defensible: the lock file is a pnpm workspace lock, the build is orchestrated by a task runner configured for that workspace, and a contributor with the wrong manager gets a fast failure instead of a subtly different dependency tree. The published package itself is built to be consumed either way, with separate module and commonjs entry points and a type declaration exported alongside them, so nothing about the artefact favours one manager.

## The readme says Python 3.9, the manifest requires 3.10

There is a small documentation discrepancy worth knowing about because it is the kind that produces a confusing support request. The requirements section, which is tucked inside a collapsed block in the readme, states Python 3.9 or higher. The manifest requires 3.10 or later, and the classifiers list only Python 3.10. So the readme is one minor version optimistic, and the manifest is the authority. That is a trivial fact on its own, and it matters only because the requirements block is hidden, so a reader skimming the page for a version number will find it easily and find the wrong one. The same block makes a second claim, that the library is compatible with both major versions of the OpenAI Python client. That claim is more interesting when read against the manifest, because the model provider's client is not a dependency at all. It appears only in the development extra, where it is used for tests. The reason is visible in the documentation rather than the manifest: the section on custom client configuration explains that you either initialise a client that every evaluator will use or pass one to a single evaluator instance. You supply the client, so the library has to work with whatever major version you brought, and taking a hard dependency on one of them would defeat the purpose. That design is the answer to the compatibility claim, and it is also the escape hatch from the gateway default, because a client pointed at your own endpoint overrides everything else.

## A development environment assembled from a task file, direnv and pinned 2022 formatters

The repository is a good example of a two-language project keeping one developer setup coherent, and the top-level files show how. A task file drives everything, and it uses the marker-file pattern rather than a script, so each step records that it has happened and later targets depend on the file rather than re-running the work. The first step removes the virtual environment and syncs the development and numerical extras with a fast Python package manager, the second syncs again for a different marker, and the third installs the pre-commit hooks. The final target does not run anything at all, it prints a line telling you to source a shell script to enter development mode, which is a polite way of saying the environment is activated by direnv when you enter the directory, with the shell script and its directory entry both committed. A single test target runs both suites, with the Python side through the fast package manager and the JavaScript side installing from a frozen lock file before running the test runner, so a contributor cannot accidentally test against a drifted dependency tree. The pinned development tools are the one thing that looks dated. The formatter is pinned to a 2022 release and the import sorter to a version from the same year, while the runtime requirement is Python 3.10 or later and the project is on its third release. That is a deliberate trade in one direction, stable formatting that does not churn diffs, and a maintenance cost in the other, since those tools will eventually stop supporting the Python versions people actually use. The newer linter is configured to check import order only and defer formatting to the older tool, which is a tidy division rather than an accident. Around it sit a type checker configuration, a pre-commit configuration, a version manager configuration for the toolchain itself, a workspace configuration for the JavaScript side, and two sets of agent instructions.

## Conclusion

AutoEvals is the right tool if your evaluations are subjective and you need to read why a grader decided what it decided, because the rationale in the result metadata is the feature and no single-number framework gives you that. The per-example workflow is the second reason, since these are templates on disk rather than functions in a library, so you can copy one, edit the prompt and re-run a single case while you work out whether the grader or the output is wrong. Three things to settle before you wire it into a pipeline. Set an explicit endpoint, because the resolution order falls through to a hosted gateway when none is configured and your prompts and completions will go there. Decide which scorers you need before you install, since the statistical ones require an extra and the multi-provider extra is separate again and is the one the all extra omits. And pin the two packages deliberately, because the Python and JavaScript lines are released under separate version tags, so the number you see in one ecosystem tells you nothing about the other.

## FAQ

### Where do I send my evaluation requests by default?

The base URL is resolved from an environment variable for an OpenAI-compatible endpoint, then from a second variable for a gateway, and if neither is set the default is a gateway hosted by Braintrust. Setting only the model provider's API key therefore routes prompts and completions through that hosted gateway, so set an explicit endpoint or pass your own client if that is not what you want.

### Why do the statistical scorers fail after a plain install?

The numeric libraries they need are declared as an optional extra rather than a base dependency, so a default install carries the templating, schema validation, edit distance and YAML pieces only. Install the extra that pulls in the numeric stack before calling the statistical scorers.

### How do I get support for model providers other than OpenAI?

There is a separate optional extra for that, and it comes with a bounded version range on its translation library. Note that the extra named for installing everything expands only to the numeric extra, so it does not include the multi-provider one and you will need a second install.

### What does an evaluation result contain?

A score on a scale from zero to one, plus metadata carrying a rationale and the raw outputs the evaluator produced. The rationale is the part to read when a score moves, and the synchronous and asynchronous entry points on the same evaluator let you choose between sequential and concurrent runs.

### How do the Python and JavaScript packages relate to each other?

They are released under separate version tags, with the most recent pair published on the same day at the same version, so the two version lines move independently. The scorer reference in the repository is the place to check what each language implements, since the two are not translations of one another.

## Sources

- [braintrustdata/autoevals on GitHub](https://github.com/braintrustdata/autoevals)
- [Issues](https://github.com/braintrustdata/autoevals/issues)
- [License: MIT](https://github.com/braintrustdata/autoevals/blob/main/LICENSE)
- [README](https://github.com/braintrustdata/autoevals/blob/main/README.md)
- [Releases](https://github.com/braintrustdata/autoevals/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/braintrustdata-autoevals