openai/evals: a YAML registry and Python harness for LLM evaluation

The problem openai/evals addresses, and who it is written for

Model upgrades break prompts. The README frames the motivation around exactly that: without evals, it says, it is difficult and time intensive to understand how different model versions might affect your use case. The repository responds with a registry of evals that test different dimensions of OpenAI models, plus the ability to write your own. There is a third category the README calls out explicitly: private evals built from your own data, which represent common LLM patterns in your workflow without exposing that data publicly. That last point matters more than the registry itself for most teams, because the public benchmarks are useful for orientation and almost never match a production prompt.

The intended audience splits in two. The first group runs existing evals locally and never touches the repository internals; the README gives them a one-line pip install. The second group authors evals, and the README suggests cloning the repository directly. Within that second group there is a further split worth noticing. The FAQ addresses the reader who says they choose not to code, and answers that following an existing eval template means providing data in JSON and eval parameters in YAML, with no evaluation code at all. That is a real lowering of the barrier, but the same FAQ paragraph warns that a good eval will inevitably require careful thought and rigorous experimentation. The tooling reduces typing, not judgement.

Two halves of the repository: a runner and a registry

The architecture is easiest to understand as two separable pieces. The first is the Python package, which executes evaluations and reports results. The second is the registry, stored under evals/registry, which holds eval definitions as YAML and the data they consume. The README notes that the registry is stored using Git-LFS and that the pointer files under evals/registry/data are populated only after fetching. That single detail explains the most common first-run confusion: a fresh clone contains small text files where datasets should be, and nothing in the directory listing makes that obvious.

Eval definitions appear to be declarative. The README points to evals/registry/evals/coqa.yaml as an example of the same evaluation implemented in multiple ways across different eval templates, which is a deliberate choice to make template differences legible by comparison rather than by prose. Datasets live as JSON. For evaluations that need more than a single model call, the README names a Completion Function Protocol, documented separately, covering prompt chains and tool-using agents. That is the extension point where a declarative YAML file stops being sufficient and Python re-enters, and it is the seam worth understanding before you commit to the framework.

Getting it running: LFS, pip, and the API key

Setup has three independent prerequisites, and skipping any one produces a different failure. First, an OpenAI API key, specified through the OPENAI_API_KEY environment variable. The README also flags the API costs incurred when running evals, which is not boilerplate: a large eval against a hosted model is a metered operation, and a registry full of benchmarks makes it easy to start one without thinking about the bill.

Second, Python 3.9 or newer, stated as the minimum required version. Third, Git LFS, installed before fetching data. From inside a local copy of the repository the README gives:

cd evals git lfs fetch --all git lfs pull

For a single eval, it offers a narrower form: git lfs fetch --include=evals/registry/data/${your eval} followed by git lfs pull. That targeted fetch is the right default when you are chasing one benchmark, since pulling every dataset to run one eval is wasted bandwidth and disk.

The installation path depends on intent. Running evals without contributing: pip install evals. Authoring evals: clone the repository and run pip install -e . so that edits take effect without reinstalling. Optional formatting hooks come from pip install -e .[formatters], after which pre-commit install wires them into git hooks, with pre-commit run --all-files for a manual pass over the repository. Results can also be logged to Snowflake if you have one, which requires four more environment variables: SNOWFLAKE_ACCOUNT, SNOWFLAKE_DATABASE, SNOWFLAKE_USERNAME, and SNOWFLAKE_PASSWORD. The README mentions a Weights & Biases route for running and creating evals as well.

The contribution freeze is the constraint that shapes the registry

The README states plainly that evals with custom code are not currently being accepted, while model-graded evals with custom model-graded YAML files still are. Read that against the rest of the document and the design intent becomes clear: the registry is meant to stay declarative and reviewable. A reviewer can read a YAML file and a JSON dataset; a reviewer cannot easily audit arbitrary Python submitted by a stranger, particularly when the eval will run against models the maintainers care about. The README adds that OpenAI staff actively review contributed evals when considering improvements to upcoming models, which is a second reason for the restriction.

The practical consequence is a split between local and shared work. You can write custom evaluation logic and run it yourself, using the completion function protocol. You cannot upstream it in its current form. Teams that treat the repository as a place to publish their evaluation methodology will find that door closed for anything beyond model-graded YAML. Teams that treat it as a local harness are unaffected. The README also notes that private evals exist precisely so that workflow-specific data need not be made public, which is the sanctioned path for proprietary evaluation sets.

Known rough edges and the case for a different tool

The README documents a hang at the very end of a run, after the final report, and advises that it can be interrupted safely, with the eval finishing immediately after. It is a small thing that is disproportionately annoying in automation: a CI job that never exits looks like a failure, and the workaround is to treat a completed report as the success signal rather than the process exit code. That is a workaround, not a fix, and the README presents it as a known issue rather than a solved one.

The larger limitation is scope. The setup path is built around an OpenAI API key. Nothing in the supplied material describes a first-class adapter for running the same registry against a locally hosted or third-party model, so if your evaluation question is which of several non-OpenAI models to use, this is the wrong starting point.

A real alternative is a general-purpose evaluation library such as EleutherAI's lm-evaluation-harness. The difference is architectural rather than cosmetic. lm-evaluation-harness is built around running a fixed set of academic benchmarks against arbitrary model backends, including local weights, and it treats the model interface as the pluggable part. openai/evals inverts that: the model provider is assumed and the evaluation definitions are the pluggable part, expressed in YAML and JSON so that non-programmers can add them. Pick based on which side of that inversion you need to vary. If you are swapping models, the harness model fits. If you are swapping tasks and want the definitions to be reviewable by people who do not write Python, this repository fits.

Maintenance cost and the licence question

The repository is not archived and the last push recorded is 2026-04-14, so it is being touched. No releases were retrieved, which means there is no tagged version to pin against in the supplied material; installation is from the repository or from the pip package, and the README does not describe a versioning or deprecation policy for eval definitions. For a registry whose YAML files are the interface, that is worth knowing before you build a pipeline around a specific eval name.

The licence field reports NOASSERTION, so the machine-readable metadata does not resolve to a standard identifier. The README's disclaimer supplies the operative statement: by contributing, you agree to make your evaluation logic and data available under the same MIT license as the repository, and you must have adequate rights to upload any data used in an eval. It further states that OpenAI reserves the right to use contributed data in future service improvements, and that contributions are subject to OpenAI's usage policies. Two consequences follow for anyone planning to contribute. First, check that you hold the rights to any dataset you intend to submit, since the burden is placed on the contributor. Second, understand that contribution is not a one-way deposit into a neutral commons. This is a description of the terms as written, not legal advice; if the data matters commercially, have someone qualified read the disclaimer against your own obligations.

Who should adopt openai/evals, and what to check first

Adopt it if your evaluation target is OpenAI models and your evaluation definitions are naturally expressible as data plus a grading rubric. The model-graded YAML path is the well-supported one, and the private-eval pattern gives you a way to keep workflow-specific data out of public view. The completion function protocol covers the cases where a single call is not enough, though that is also where you leave the declarative surface and start writing code that the registry will not accept.

Do not adopt it if you need to compare non-OpenAI or self-hosted models through the same registry, or if your core evaluation logic is custom Python that you expect to share upstream. In both cases the constraint is structural rather than a matter of configuration.

Three things to verify before you build on it. Confirm that git lfs fetch --all finishes and that evals/registry/data contains real datasets rather than pointer files, because everything downstream depends on that step. Confirm your interpreter is at least Python 3.9. Then run one existing eval end to end and observe the documented hang after the final report, so that when it appears in your own pipeline you recognize it as the known issue and not as a regression you introduced. The eval that answers your actual question is the one you write yourself; the registry is where you learn the shape of a good one.

Editorial conclusion

Adopt openai/evals if you want to run or author YAML-defined, model-graded evaluations against OpenAI models and you accept Git LFS as a prerequisite for the registry data. Do not adopt it if you need to evaluate non-OpenAI models, since the README's setup path assumes an OPENAI_API_KEY, or if your eval logic requires custom Python, which the README states is not currently being accepted. Before committing, verify three things: that git lfs fetch --all completes and populates evals/registry/data, that your Python is 3.9 or newer, and that you can run one existing eval end to end, including the known hang after the final report.

Official sources

1. Issues
2. openai/evals on GitHub
3. README