Inspect Evals is a shared library of model evaluations, and a policy problem as well

A catalogue of evaluations, not a framework

Inspect Evals is a library of evaluations built using Inspect AI, and the readme points you straight at the documentation site with the instruction to explore evaluations and read the docs. The two things the project asks you to do are run an eval and add an eval to the Inspect Evals Register, the second of which means adding a listing that points to your own implementation and documentation. That second item is the interesting one, because it distinguishes the catalogue from a repository. A repository of evals is a pile of code. A catalogue with a register is a way of pointing at implementations that may live anywhere, which is what makes it useful to someone who wants to know which evaluation exists for a capability rather than which implementation is vendored here. The primary language is Python, the licence is MIT, and the releases are frequent, with v0.25.0 on 2026-10-17 after v0.24.0 and v0.23.0 in the two weeks before it.

Who runs it, and why that matters for trust

Inspect Evals is maintained by Generality Labs, described as a London-based nonprofit which builds tools for evaluating risks and mitigations in frontier AI, and the project was founded in 2024 with contributions from the UK AI Security Institute, Arcadia Impact and the Vector Institute. The provenance matters because an evaluation catalogue is infrastructure for claims about model behaviour, and who curates it determines whose framing of a capability is treated as the default. The readme is careful about the other side of that, though, because there is a formal register for outside implementations and a request for inquiries, suggestions, or expressions of interest through a form. The maintainers are listed separately in a MAINTAINERS file, and the repository carries a NOTICE file and a directory for third-party licenses, which is what you would expect from a catalogue that mixes code from many sources.

The contribution policy is the unusual part

The readme carries an important notice about the contribution policy, and it is worth reading before you plan a pull request. The project has updated its contribution policy to only accept pull requests from pre-approved contributors. If you identify an issue with an evaluation, the instruction is to raise an issue rather than to fix it in a pull request, and to upload .eval logs to the log uploader, which demonstrate the problem. The maintainers will review the logs to better understand the severity and the cause of the issue. That is a defensible policy for an evaluation catalogue, because a broken or subtly altered evaluation can quietly change a headline number, and the log file is the evidence that lets a maintainer see what actually happened rather than reading a description of it. The trade-off is real, and it is that the project is less open to drive-by contributions than a normal repository, which is why the register and the issue route matter more here than they would elsewhere.

Python versions, and the honest note about 3.14

The recommended Python version is 3.11 or 3.12, and you should be able to run all evals on those versions and develop the codebase without any issues. You can pin a specific version with uv.

uv python pin 3.11

The note about newer versions is unusually specific and worth reading twice. As for Python 3.13, you should be able to run all evals except sciknoweval, whose dependency is gensim and which currently does not support 3.13 and later, and development should work under 3.13 but is relatively untested. As for Python 3.14, at the time of writing many packages have yet to release versions for it, so it is unsupported, and the major one used by some Inspect Evals is torch. And there is an invitation rather than a dead end, because if you find running uv sync succeeding on 3.14, let the maintainers know and they will remove the paragraph. That is the right way to handle a moving target, since the constraint is a fact about the dependency tree rather than a policy.

Running an evaluation takes one command

There are two ways of using Inspect Evals, as a dependency from PyPI in your own project, and as a standalone checked out repository. From PyPI the install is the ordinary one.

pip install inspect-evals

From a checkout, the dependencies are installed with uv sync, and then the evaluations run through the inspect command.

uv run inspect eval inspect_evals/arc_easy --model openai/gpt-5-nano
uv run inspect eval inspect_evals/arc_challenge --model openai/gpt-5-nano

To run multiple tasks at once there is an eval-set command.

uv run inspect eval-set inspect_evals/arc_easy inspect_evals/arc_challenge

And the tasks can also be imported as ordinary Python objects.

from inspect_ai import eval, eval_set
from inspect_evals.arc import arc_easy, arc_challenge
eval(arc_easy)
eval_set([arc_easy, arc_challenge], log_dir='logs-run-42')

The logs are then read back with the inspect view command, or in VS Code with the Inspect AI extension for viewing logs. The model can be pinned once in a .env file in the working directory with the INSPECT_EVAL_MODEL variable and the API key, and Inspect supports many providers including OpenAI, Anthropic, Google, Mistral, Azure AI, AWS Bedrock, Together AI, Groq, Hugging Face, vLLM and Ollama.

The hardware guidance is specific because the datasets are large

The disk requirements section is the kind of detail that saves a wasted afternoon. The recommendation is at least 35 GB of free disk space, because the full installation takes about 10 GB and you also need space for the uv cache and the datasets cache, most of which are small but some take 13 GB, with MMIU named as the example. Running some evals, and CyBench and the GDM capabilities evals are named, may require extra space because they pull Docker images, and the recommendation there is at least 65 GB of extra space for evals that have Dockerfiles in their file tree, on top of the 35 GB. The total advice is to be comfortable running evals with 100 GB of free space, with a note that if you run out of space while having 100 GB or more available, that might be a bug. Datasets and other large assets are cached under the platform cache directory, which is ~/.cache/inspect_evals on Linux and ~/Library/Caches/inspect_evals on macOS, and the INSPECT_EVALS_CACHE_DIR environment variable moves it.

The repository is built like a monorepo with a lint preset

The top level of the repository shows how seriously the process is taken. There is src/ for the code, packages/ for the isolated eval packages, register/ for the register entries, docs/ for the documentation, tests/, tools/ with the generator scripts, adr/ for architecture decision records, internal/, analysis/, agent_artefacts/ and third-party-licenses/. The documentation is generated, with an automatically generated usage section in the readme between markers and a script that creates missing readmes, and there is an ASSETS.yaml with a script that generates the asset manifest and a check that evals are listed. The Makefile check target is a pipeline of steps, and it is worth reading because it shows what the project considers non-negotiable: ruff format, ruff check with fix, mypy over src and tests, generate_readmes, generate_asset_manifest, check_unlisted_evals, check_changelog and uv lock --check. There is a separate lint tool with a monorepo preset and a list of exclusions for vendored and sandbox code, including files that are not even valid Python. The version policy lives in its own files, with TASK_VERSIONING.md and PACKAGE_VERSIONING.md, and a changelog.d/ directory for the fragments.

Where this is the wrong choice

Three limits are worth naming. The first is that the catalogue is for reproducing published evaluations, not for measuring a capability of your own product, so a team looking for a benchmark of its own design will find infrastructure here but not a template. The second is that the contribution policy, sensible as it is, means the project cannot take improvements from the people who have them, and a bug in an evaluation has to be reported with logs rather than fixed, which slows the feedback loop for the people most likely to notice a problem. The third is the cost of running it. A hundred gigabytes of disk, Docker images for some evaluations, and a model provider key per run is a heavy footprint for what looks like a pip install, and an organisation that already runs its own harness may reasonably decide that the shared catalogue is a cross-check rather than a production dependency. What the project gives in return is the thing that is genuinely hard to build alone, which is a consistent, reproducible way to run a wide set of public benchmarks and compare the logs.

Editorial conclusion

Inspect Evals suits a team that wants to run several public benchmarks against a model with one tool and one log format, and it suits anyone who needs to compare an eval result with the published numbers. It is the wrong tool if you need an evaluation for your own internal capability, because the catalogue is for reproducing published results rather than inventing new ones. It is also the wrong choice if you expected to open a pull request freely, since the contribution policy accepts them only from pre-approved contributors. Before contributing, read the approved contributors list and the evaluation checklist, because the checks are automated and a pull request that skips them will fail on the same rules. The last push was on 2026-10-19 and the current release is v0.25.0 from 2026-10-17.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. UKGovernmentBEIS/inspect_evals on GitHub