Inspect_ai: version from git tags, web UI from a submodule, dependencies in five files
Project brief: Inspect: A framework for large language model evaluations. Inspect provides many built-in components, including facilities for prompt engineering, tool usage, multi-turn dialog, and model graded evaluations.
At a glance
- What is it?
- Inspect is the UK AI Security Institute's framework for large language model evaluations, and its own source is more instructive than its feature list: the version is derived from semver-shaped tags in a repository with no releases, the web interface arrives as a prebuilt bundle whose TypeScript source lives in a submodule hosted elsewhere, and every dependency pin carries a note about which upstream release broke it.
- Who is it for?
- Inspect suits teams that want evaluation code they can read, pin and extend, and that treat an eval harness as something to own rather than rent from a vendor.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 22, 2026, and from our analysis. They are not legal advice.
Star history
- UKGovernmentBEIS/inspect_ai3,308 ★ · +877
| Repository | 2026-07-30 | 2026-11-22 |
|---|---|---|
| UKGovernmentBEIS/inspect_ai | 2.4k | 3.3k |
877 Stars in the last 115 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
The version is read off git tags and the repository publishes no releases
The build backend is setuptools with setuptools_scm, so no version number is written into a file. The configured describe command runs git describe with --dirty, --tags and --long, plus a --match pattern restricted to [0-9]*.[0-9]*.[0-9]*, which means only tags shaped like three numeric components take part in the calculation and anything else is invisible to it. The repository has no GitHub releases at all, which leaves the tag list as the only version record it keeps, with a CHANGELOG.md at the root serving as the human-readable history instead. The last push on the default branch main is dated 2026-10-03. For anyone who wants to pin an exact version, that combination decides what can be pinned and what the pin resolves to.
The web UI ships prebuilt while its TypeScript source lives elsewhere
The web interface lives in a git submodule at src/inspect_ai/_view/ts-mono/, and those steps apply only to people touching the TypeScript and React frontend, since Python-only contributors can skip them. The one-time setup instructions are not written here: they point at a guide hosted in a different repository, meridianlabs-ai/ts-mono. Packaging keeps the same split. pyproject.toml excludes _view/ts-mono* from the package data, with a comment explaining that the git-tracked files are pruned in MANIFEST.in and that the viewer ships prebuilt as _view/dist. So the frontend source is in neither the file tree nor the wheel, and the compiled bundle is what an installed package actually carries. Two lockfiles at the root, uv.lock and yarn.lock, advertise the two toolchains involved.
Dependencies live in five requirements files that pyproject only points at
The manifest delegates its dependency list to plain text files. The dynamic section sets dependencies from requirements.txt, and four optional groups read requirements-dev.txt, requirements-doc.txt, requirements-dist.txt and requirements-math.txt, all five sitting at the root beside pyproject.toml. Development therefore starts as an ordinary editable install with the dev group:
git clone https://github.com/UKGovernmentBEIS/inspect_ai.git
cd inspect_ai
pip install -e ".[dev]"The uv path is offered as an alternative, with the emphasis that it is supported but not required:
uv sync --extra devuv.lock records a reproducible development resolution, and the instruction for changing anything is to edit the appropriate requirements file and refresh the lockfile rather than reach for uv add.
Pin comments record which upstream releases broke the build
The requirements file carries its own history in trailing comments. click is held at 8.1.3 or newer with 8.2.0, 8.2.2, 8.3.0 and 8.3.1 excluded, because those releases break optional flag values according to pallets/click#3084, with a note that the problem is fixed in 8.3.2. 8.2.1 is not excluded, so the exclusions track specific bad builds rather than a refusal of the whole 8.2 and 8.3 series. anyio is floored at 4.14.0 for an asyncio Lock and Semaphore waiter deadlock after cancellation. fsspec is capped from above at 2026.6.0 to stay aligned with hf datasets. Two pins are conditional on the interpreter, exceptiongroup below Python 3.11 and zipfile-zstd below 3.14, and zipp is held at 3.19.1 with a comment saying it is not directly required and is pinned by Snyk to avoid a vulnerability.
make check runs lint, types and a suppression gate but never the tests
The Makefile builds check from three targets in order: ruff, mypy and suppressions-check. The ruff recipe runs check --fix and then format at the root, and repeats both inside src/inspect_tool_support/src, a second source tree that the top-level file list does not show. The mypy recipe covers src and tests while excluding tests/test_package. suppressions-check calls python3 .github/scripts/check_suppressions.py, and a sibling target passes --update to regenerate suppressions.json, the root file that the gate exists to police. Tests are separate: test runs plain pytest, and test-parallel runs pytest -n auto. The file ends by including docs/evals/inspect-evals.mk, so the evaluation material brings its own makefile fragment, and the README adds that these commands take a uv run prefix inside a uv-managed environment.
The lint selection is short with two rule families switched off in place
The ruff configuration selects E and W from pycodestyle, F from flake8, D from pydocstyle, I from isort, B from bugbear, SIM101 for duplicate isinstance, DTZ for timezone handling and TID251 for banned API calls. Two more families appear commented out in the same list, RET for return style and RUF for the wider ruff rules, so they are switched off deliberately rather than left unconfigured, and docs/ is excluded from linting altogether through extend-exclude while src is named as the source root. Around that sit the editor hooks: a .pre-commit-config.yaml at the root, a make hooks target that runs pre-commit install, and a note that opening the project in VS Code prompts for the Python, Ruff and MyPy extensions.
Two assistant instruction files sit next to the contributor and security policies
The root holds AGENTS.md and CLAUDE.md alongside CONTRIBUTING.md and SECURITY.md, with .agents/ and .claude/ directories as well, so guidance for two different assistant tools is versioned in the same tree as the human-facing contributor and security policies. The same block of the project page points coding agents at documentation in text form: a structured index at inspect.aisi.org.uk/llms.txt, the user guide concatenated as Markdown at llms-guide.txt, and llms-full.txt bundling the API and CLI reference together, with the rule that any single page can be fetched as Markdown by appending .md to its .html path. Documentation for humans is built differently, from the doc group and quarto, with requirements-doc.txt holding that group.
Two hundred pre-built evaluations and an examples tree that names the capabilities
The count of ready-made evaluations is over 200, collected at inspect.aisi.org.uk/evals/ and described as runnable on any model. The examples directory is more specific about what the framework can drive: approval/, ask_user/ and human/ for people inside a run, browser/, computer/ and code_execution.py for agents acting on systems, mcp_tools.py and remotemcp.py for tool use, hooks/, intervention/ and inline_cards/ for shaping a run, prefill.py, early_stopping.py and cache.py for control over generation, evalset.py, evals_in_eval/ and categorical_demo.py for composing evals, and reasoning.py, popularity.py and biology_qa.py as task examples, with bridge/, responses-bridge.py and http_proxy/ beside them. The wheel also carries binaries, a sandbox tools version file with SHA256SUMS, and the same pair under util/_restic.
Editorial conclusion
Inspect suits teams that want evaluation code they can read, pin and extend, and that treat an eval harness as something to own rather than rent from a vendor. Before installing it, check three things against the repository rather than the landing page: which tag you are actually getting, since no GitHub releases exist and setuptools_scm only counts tags shaped like three numeric components, and the last push was 2026-10-03; whether you need the web interface at all, because its TypeScript source is a submodule from another repository and the wheel ships a prebuilt bundle instead; and whether the exclusions in the requirements file still make sense for your Python version, since the click and anyio pins each name a specific upstream bug. Python-only contributors can skip the frontend entirely and work from the editable install with the dev group.
Frequently asked questions
What is inspect ai, the UK AI Security Institute framework?
It is a framework for large language model evaluations with built-in components for prompt engineering, tool usage, multi-turn dialog and model graded evaluations. Extensions for new elicitation and scoring techniques can come from other Python packages, and a collection of over 200 pre-built evaluations is published alongside it.
How do I use inspect ai for development?
Clone the repository, install it editable with the dev group, and drive the work through make check for lint, types and the suppressions gate, then make test, or pytest -n auto through test-parallel. In a uv-managed environment those commands take a uv run prefix, and uv sync --extra dev is offered as an alternative to the editable install.
Why does inspect_ai have no GitHub releases?
The version is derived by setuptools_scm from the tag list rather than from a number in a file, and the configured describe command matches only tags shaped like three numeric components. With no releases published, tags are the only version record, and CHANGELOG.md at the root carries the human-readable history instead.
Where does the inspect_ai web interface source live?
In a git submodule at src/inspect_ai/_view/ts-mono/, with the one-time setup guide hosted in the meridianlabs-ai/ts-mono repository. The wheel excludes that path and ships the viewer prebuilt as _view/dist, and the root carries a yarn.lock next to uv.lock for the TypeScript side.
Does inspect_ai pin its dependencies for reproducibility?
uv.lock records a reproducible development resolution, but dependencies are still declared in requirements*.txt and exposed through pyproject.toml, with dev, doc, dist and math groups in separate files. The guidance is to update the requirements file and refresh the lockfile rather than rely on uv add, and individual pins carry comments naming the upstream breakage they avoid.
Official sources
Where developers are discussing it
Posts on Hacker News, dev.to and Lobsters that link to this repository, found by our scan of those communities. 0 in the last 30 days; newest first.
- 17,600 actions, no human at the wheel: how to start red teaming AI agents in 2026dev.to · Oct 14, 2026 · 5 points · 1 comments
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/ukgovernmentbeis-inspect-ai)Star history chart for your README
[](https://hysenlabs.com/projects/ukgovernmentbeis-inspect-ai)Was this useful?