VeriRun treats a benchmark number as something that must be provable

The runtime under the benchmark, not another benchmark

VeriRun is infrastructure for running code and agent evaluations where the results have to be reproducible, recoverable, attributable and safely isolated. Its README draws the distinction itself: this is not another leaderboard, it is the runtime underneath trustworthy executable evaluation.

The framing is the most useful part of the project. Open-source benchmark harnesses answer a workload question, namely how to run a given benchmark. Running executable evaluation at scale raises a different set of problems, and the README lists them precisely: model calls are asynchronous, rate limited, expensive and cancellable; generated programs are untrusted; task runtimes have a long tail; retries can silently duplicate work; the benchmark, prompt, tests, model and runner versions all move the score; and a model failure, a verifier defect, a sandbox failure and a scheduler failure must not be indistinguishable in the output.

That last item is the one most evaluation setups get wrong. When infrastructure trouble is recorded the same way as a model getting the answer wrong, the resulting number is not a measurement of the model.

The audience is teams running evaluation as an ongoing obligation rather than a one-off, and anyone computing rewards from executed code for post-training.

Six invariants, stated as targets rather than guarantees

The project organises itself around six invariants, and they are worth reading because each names a specific failure.

Immutable provenance means every result records the benchmark, prompt, candidate, tests, verifier image, model revision, sampling configuration and runtime policy behind it. Replay before claims means frozen candidates can be re-verified without calling the model, with replay differences classified rather than hidden. Structured failure semantics keeps compile errors, test failures, timeouts, memory exhaustion, policy violations and infrastructure failures distinct. Isolation is treated as evidence rather than a checkbox, requiring an explicit threat model and an attack regression suite on a named environment. Execution is at-least-once while results are effectively-once, with idempotent commit preventing a retry producing two final answers. And model capability is reported separately from infrastructure quality, with unreliable runs marked partial or invalid rather than published as conclusions.

Then comes the sentence that makes the list trustworthy: these are design targets until the corresponding roadmap gate is completed and linked to reproducible evidence.

Stating that a principles list is aspirational, inside the principles list, is rare and it is the correct thing to do. It converts the section from a claim into a plan, and it means a reader can check progress against gates rather than assuming the guarantees hold.

The evidence directory is the claim made structural

An evidence-first project earns the label through its build, and this one does. The repository holds an evidence directory, and its task file defines a long list of evidence-producing targets covering containers, the control plane, distributed execution, concurrency, fault injection, reliability, synthetic workloads, a specific benchmark and Kubernetes.

That arrangement means generating evidence is a build target rather than an activity someone remembers to perform. The difference matters: a claim backed by a command anyone can run is checkable, while a claim backed by a paragraph is not.

Supporting documents follow the same pattern, with a benchmark protocol, a roadmap, a changelog, a security policy, a code of conduct and a contributing guide all present as separate files. A benchmark protocol as its own document is unusual and appropriate here, since the project's entire argument is that how you run the benchmark determines whether the number means anything.

The architecture is described as delivered incrementally, with the README stating plainly that the components in its diagram are not all implemented today. The shape is a control plane owning intent and durable state, an execution plane performing retryable attempts, an orchestrator distributing work, a sandbox manager placing untrusted code in isolated jobs, and adapters that preserve upstream benchmark semantics rather than reimplementing them.

The warning at the top is the thing to act on

Before any of the design merits, the README carries a notice that governs safe use, and it deserves repeating rather than summarising.

The project is pre-alpha. The local executor from the earliest version remains present for trusted fixtures only, is explicitly not a security boundary, and must not be used for model-generated or otherwise untrusted code.

Read that as written. The component most convenient to reach for, because it runs without a cluster, is the one that provides no isolation. Executing model-generated code without a boundary on a machine you care about is how an evaluation harness becomes an incident, and the presence of a convenient unsafe path next to an inconvenient safe one is exactly the situation where people take the convenient path.

The safe path carries real infrastructure requirements, since the isolation claims are tied to a named environment combining Linux, a container orchestrator and a sandboxed runtime. That is not something to stand up casually, and it is the honest cost of the guarantee.

The interpreter requirement is narrow too, pinned to a single minor version, with the build dependencies pinned to exact versions in the same file. Reproducibility taken seriously produces constraints like these, and it means a dedicated environment rather than a shared one.

Where it sits today

Version 0.6.0 was published on 2026-09-16, the same day as the last push, and the release is described as delivering an evidence contract with tracing correlation across the durable control-plane boundary.

A project at 0.6 with an explicit pre-alpha label and a roadmap gating its own guarantees is not something to build a reporting pipeline on this quarter. The invariants that would justify adopting it, particularly the isolation and provenance guarantees, are the ones still tied to gates.

There is also an inherent scope cost. Running this properly means operating a control plane, a database, a distributed execution framework and a sandboxed container environment, which is a platform rather than a tool. For a team already running that infrastructure the marginal cost is modest; for a team that is not, the evaluation runtime brings a cluster with it.

Finally, the benchmarks it adapts are upstream projects with their own semantics, which the README says the adapters preserve rather than reimplement. That is the right decision and it means your results remain tied to those upstream projects' behaviour and versions, which is precisely what the provenance requirement is designed to record.

Scripting a harness yourself is the alternative

The alternative nearly everyone is using is running a benchmark harness directly, with scripts around it for sampling, retries and collecting results.

The difference is what happens when something goes wrong at scale. A script-driven harness answers the workload question well and leaves every operational concern to you: a retry may duplicate work, a sandbox failure may be recorded as a wrong answer, a rerun after a model update may not be comparable to the previous one, and the record of which prompt and test version produced a given score may exist only in a filename. For a single evaluation run reported once, none of that matters much.

It matters when the number is used repeatedly, compared across versions, or fed into training as a reward, because then an unnoticed infrastructure failure becomes a bias in a training signal rather than a bad row in a table.

Take the harness and scripts when you are running an evaluation to answer a question now. Take an approach like this one when evaluation is continuous, when results inform decisions weeks later, or when generated code must be executed under a boundary you can describe to a security reviewer. Given the pre-alpha status, the realistic move today is to read the invariants and the benchmark protocol and apply them to your own setup, whether or not you adopt the runtime.

Apache terms and the first thing to check

VeriRun is Apache-2.0 licensed with the file present and declared in the project manifest, which includes an express patent grant and permits commercial use. This is not legal advice, though it is the permissive choice least likely to complicate an internal review.

The manifest fixes the interpreter to one minor version and pins its build dependencies exactly.

requires-python = ">=3.12,<3.13"

That narrowness is consistent with the project's reproducibility goals and means a dedicated environment.

The order to approach this in is clear from the README itself. Read the notice about the local executor first and decide whether you can stand up the isolated path, because the convenient component is explicitly not a security boundary and running model-generated code through it is the mistake the project warns against on its first screen. Then read the roadmap to see which invariants have passed their gates, since the guarantees are described as targets until they have, and that mapping is what tells you whether the runtime does today what you need it to do.

Editorial conclusion

VeriRun is aimed at teams for whom evaluation is continuous rather than occasional, where a score gets compared across model versions or fed back as a training reward and an unnoticed infrastructure failure would become a bias rather than a bad row. It is not ready to carry that load yet, since the project labels itself pre-alpha and states its six invariants remain design targets until their roadmap gates are met. Read the notice about the local executor before anything else, because it is explicitly not a security boundary and must not run model-generated code, and check the roadmap to see which invariants have passed their gates, since that mapping is what tells you what the runtime actually guarantees today.

Frequently asked questions

Official sources

1. aaron-for-value/VeriRun on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases