Inspect: A UK AI Security Institute Framework for Structured LLM Evaluations

What Inspect Is For and Who It Serves

Inspect is a Python framework for large language model evaluations, created by the UK AI Security Institute. It targets teams that need to run systematic, repeatable evaluations of LLMs, not just ad-hoc prompts. The framework provides built-in components for prompt engineering, tool usage, multi-turn dialog, and model-graded evaluations. That scope makes it useful for safety researchers, model developers, and organizations that need to benchmark models against specific tasks. The README describes it as a framework, which implies a structured approach: you define evaluations as code, using Inspect's abstractions, rather than writing one-off scripts. The intended user is someone who wants a shared, extensible foundation for evaluation work, not someone who just wants to call a model API and print a score.

The Architecture: Components, Extensions, and Pre-Built Evals

The README does not expose the full internal architecture, but it does reveal the key design choices. Inspect is built around components: prompt engineering, tool usage, multi-turn dialog, and model-graded scoring are all provided as first-class features. Extensions can be added through separate Python packages, which means you can plug in new elicitation or scoring techniques without forking the core. The framework also includes a collection of over 200 pre-built evaluations that you can run on any model. That is a significant asset, since it gives you a starting point for common evaluation tasks. The documentation is published online, and notably, the README points to machine-readable versions: a structured index at llms.txt, a concatenated user guide, and a full bundle with API and CLI references. This suggests the project cares about making its docs accessible to coding agents, which is a practical touch for an evaluation framework.

Getting It Running: Commands and Configuration

The README gives clear setup instructions. For development, you clone the repository and install with pip: `git clone https://github.com/UKGovernmentBEIS/inspect_ai.git`, then `cd inspect_ai`, then `pip install -e ".[dev]"`. If you use uv, you can sync the development environment from the checked-in lockfile with `uv sync --extra dev`. The uv workflow is supported but not required. The project declares dependencies in `requirements*.txt` and exposes them through `pyproject.toml`. The README explicitly says that when changing dependencies, you should update the appropriate requirements file and refresh the lockfile, rather than relying on `uv add`. This is a deliberate workflow choice: it keeps the requirements files as the source of truth, even though uv is offered as a convenience. For linting, formatting, and tests, you run `make check` and `make test`. In a uv-managed environment, prefix those with `uv run`. There are also optional dependency groups: `[doc]` for documentation work, which requires Quarto to build the docs. The README does not show how to run an evaluation or configure a model provider, so that part is left to the documentation.

The TypeScript Frontend: A Submodule with Hidden Complexity

A notable detail in the README is the web UI, which lives in a git submodule at `src/inspect_ai/_view/ts-mono/`. The README warns that these steps are only needed if you plan to work on the TypeScript/React frontend, and Python-only contributors can skip this entirely. That is a clear sign of a split architecture: the core evaluation logic is Python, but the visualization layer is a separate TypeScript project. This introduces a real maintenance cost. Git submodules are notoriously finicky to keep in sync, and the README points to an external guide for the one-time setup. If you are a Python developer who wants to use the framework, you can ignore this, but if you need to customize the web UI, you are now dealing with two codebases, two toolchains, and a submodule that can go stale. The README does not explain how the frontend communicates with the Python backend, so the integration details are undocumented in the material we have.

Documentation as a First-Class Deliverable

The README puts unusual emphasis on documentation. It links to the main docs site, and then it provides a structured index for coding agents at `https://inspect.aisi.org.uk/llms.txt`. There is also a concatenated user guide as Markdown at `llms-guide.txt`, and a full bundle that includes the API and CLI reference at `llms-full.txt`. You can even append `.md` to any `.html` path to get the Markdown version of a page. This is a practical approach for developers who want to grep the docs or feed them into an LLM-assisted workflow. For a framework that is itself about evaluating LLMs, this dogfooding is coherent. However, the README does not state how up-to-date these generated docs are, or whether the llms.txt index is automatically maintained. The existence of these files suggests a mature documentation pipeline, but the material does not confirm its freshness.

Limitations and When It Is the Wrong Tool

The README reveals several limitations. First, the framework is not a lightweight library. It has a development setup with optional dependencies, a lockfile, and a frontend submodule, which implies a non-trivial footprint. If your evaluation task is a single prompt comparison, Inspect is overkill. Second, the framework appears to target structured evaluations, but the README does not explain how to define a custom evaluation or how the scoring works beyond mentioning model-graded evaluations. That means you will need to consult the documentation to get anything done, and the learning curve could be steep for teams without prior experience in evaluation frameworks. Third, the project is under active development by the UK AI Security Institute, but there are no release notes or version information in the provided material. The last push is unknown, and no recent releases are listed. This is a risk if you need stability: you might be adopting a moving target. The README also does not mention any guarantees about backward compatibility, so upgrading could break your evaluation definitions.

Alternatives and How They Differ

The README does not name any alternatives, but the obvious comparison is with other LLM evaluation frameworks like OpenAI Evals or LangChain's evaluation modules. OpenAI Evals takes a similar approach: you define evaluations in YAML or Python, and it supports model-graded scoring. The key difference is that OpenAI Evals is tightly coupled to OpenAI's API, whereas Inspect is model-agnostic, as evidenced by the claim that pre-built evaluations run on any model. LangChain's evaluation tools are part of a larger agent framework, so you get evaluation as a side feature, not a dedicated focus. Inspect, by contrast, is a standalone framework built specifically for evaluations, which means its abstractions are likely more purpose-built. However, the README does not provide a feature-by-feature comparison, so the practical differences in scoring flexibility or tool integration remain unverified. If you are already invested in LangChain, adding Inspect could mean maintaining two evaluation pipelines, which is a real cost.

Maintenance, Licensing, and Upgrade Considerations

The project is licensed under MIT, which is permissive and allows commercial use, modification, and redistribution with attribution. That is a low-friction license for most organizations, but it comes with no warranty, so you are responsible for verifying the framework's behavior. The README describes a development workflow with `make check` and `make test`, which suggests the project has its own quality gates. For your own maintenance, you will need to track upstream changes, and the lack of release information in the material makes it hard to plan upgrades. The README also mentions that extensions can be provided by other Python packages, which means you can build your own evaluation techniques, but that also means you are responsible for keeping those extensions compatible with core updates. The uv lockfile provides a reproducible development environment, but it does not guarantee that the framework's API is stable across versions. The documentation is extensive, but you will need to verify that it matches the version you actually install.

Editorial conclusion

Adopt Inspect if you are building or running structured LLM evaluations and want a framework that handles prompt engineering, tool use, multi-turn dialog, and model-graded scoring out of the box, especially if you can benefit from the 200+ pre-built evaluations. Do not adopt it if you need a minimal, dependency-light evaluation script or if you cannot accommodate the learning curve of its component model and the need to keep up with a fast-moving project. Before adopting, verify that the current documentation matches your intended elicitation and scoring techniques, check the state of the TypeScript frontend submodule if you plan to use the web UI, and confirm that the MIT license and the UK AI Security Institute's governance fit your project's compliance requirements. The framework's real test is whether its abstractions simplify your evaluation pipeline or just add another layer to debug.

Official sources

1. Official documentation
2. Official README
3. Project repository