evmbench: a detect-only benchmark harness for smart contract bug hunting

What evmbench is for, and who it is aimed at

The README states the project is "a benchmark and agent harness for finding and exploiting smart contract bugs." Those are two jobs in one repository. The harness part is an end-to-end pipeline: you upload a zip of contract files, pick an agent, and get back a structured vulnerability report rendered in a browser UI. The benchmark part is the tie-in to OpenAI's detect evaluation, whose code is pinned as a git submodule under frontier-evals/. The repository describes itself as a companion interface to that evaluation.

The audience is narrow and technical. You need to be comfortable building Docker images, running a compose stack, and reading a Codex prompt file. The README gives no hosted demo, no signup flow and no pricing, and there are no releases retrieved for the repository. The last push to main was on 2026-06-24, so the code has moved within the last few months but there is no versioned artifact to pin against. If you want a scanner you point at a mainnet address, this is not that.

The architecture: FastAPI, RabbitMQ, and a worker that runs Codex

The README publishes a full architecture diagram, and it is worth reading literally because it explains most of the operational cost. A Next.js frontend talks to a FastAPI backend on port 1337. Job state lives in PostgreSQL on 5432. Queued work goes through RabbitMQ on 5672. A component called the Instancer consumes the queue and starts a worker, using a Docker backend locally or an optional Kubernetes backend. The worker fetches its secret bundle from a Secrets Service on 8081, unpacks the uploaded zip into audit/, and runs Codex in what the README calls "detect-only" mode. Results go to a Results Service on 8083, and the frontend polls job status and renders the report with file navigation and annotations.

The end-to-end flow is documented in six steps. The UI sends the archive, the selected model key and optionally an OpenAI API key to POST /v1/jobs/start. The backend writes a job record, stores a secret bundle, and publishes to RabbitMQ. The worker then uses three files that matter if you want to change behaviour: the prompt at backend/worker_runner/detect.md, copied to $HOME/AGENTS.md inside the container, the model map at backend/worker_runner/model_map.json, which translates UI model keys into Codex model IDs, and the command wrapper backend/worker_runner/run_codex_detect.sh. The agent writes submission/audit.md, and the worker validates that the output contains parseable JSON of the form {"vulnerabilities": [...]} before uploading.

That validation step is the most interesting design decision in the repository. The worker does not grade the findings, it only checks that the agent produced machine-readable output. Everything downstream of that check is presentation. If you were hoping the harness tells you whether a reported vulnerability is real, the README does not claim it does.

Installing evmbench locally and running a first job

The README's quickstart requires Docker and Bun. It is explicit that the base and worker images must be built before the stack starts, so skipping this step will leave the Instancer with nothing to launch.

cd backend
docker build -t evmbench/base:latest -f docker/base/Dockerfile .
docker build -t evmbench/worker:latest -f docker/worker/Dockerfile .

With the images in place, copy the environment template and bring up the backend stack. The README notes that the placeholder secrets in .env.example are sufficient for local development, and that internet-exposed deployments should replace them with strong values.

cp .env.example .env
docker compose up -d --build

The frontend is a separate dev server. The Makefile offers the same two steps as targets, frontend-install and frontend-build, if you prefer make over typing them.

cd frontend
bun install
bun dev

Two URLs should respond: the frontend at http://127.0.0.1:3000 and the backend config endpoint at http://127.0.0.1:1337/v1/integration/frontend. The README does not describe what that config endpoint returns, so treat a successful response as a liveness check rather than a schema contract. From the UI, upload a zip of contract files, choose a model key, and supply an OpenAI key. The job appears in the history view and the report renders when the worker finishes.

Security posture and the BYOK trade-off

The README is blunt about the trust model: evmbench runs an LLM-driven agent against uploaded, untrusted code, and the worker runtime (filesystem, logs, outputs) should be treated as an untrusted environment. SECURITY.md is cited as holding the full trust model and operational guidance. That framing matters more than it might seem. Codex inside the worker can read the uploaded source and write files; the prompt file is copied in as AGENTS.md, which means the agent's instructions live inside the same container as the code it is analysing.

Credential handling comes in two modes. Direct BYOK is the default: the worker receives a plaintext OpenAI key through OPENAI_API_KEY or CODEX_API_KEY. Proxy-token mode is optional and keeps the plaintext key outside the worker; the worker gets an opaque token and routes through oai_proxy on port 8084. Enabling it means copying .env.example to .env, setting BACKEND_OAI_KEY_MODE=proxy and OAI_PROXY_AES_KEY, and starting compose with the proxy profile.

cd backend
cp .env.example .env
# set BACKEND_OAI_KEY_MODE=proxy and OAI_PROXY_AES_KEY=...
docker compose --profile proxy up -d --build

The README does not document key rotation, bundle expiry or what happens to a bundle after a job completes, so if you are running this on client code you should read SECURITY.md before assuming the default is appropriate. The default is convenient and the proxy mode is more work; the project presents both without recommending one.

Runtime limits and where evmbench is the wrong tool

Worker runtime is bounded by default, and the README gives one override: EVM_BENCH_CODEX_TIMEOUT_SECONDS, with a default of 10800 seconds. Three hours per audit is generous for a small contract and tight for a large protocol, and the README does not describe what the worker does when the timeout fires beyond the fact that the runtime is bounded. There is no documented retry, resume or partial-result path, and no documented rollback. The optional prunner component is described in the repo layout as "optional cleanup of stale workers", which suggests stale workers are an expected state rather than an anomaly.

The larger limitation is scope. This is detect-only. The README describes the agent writing submission/audit.md and the worker checking for parseable JSON; it does not describe exploitation, proof-of-concept generation or severity scoring, despite the project description mentioning finding and exploiting bugs. The base image bundles foundry, slither, node and other tools, so the environment could support more, but the documented flow stops at a report.

A second constraint is the dependency surface. Postgres, RabbitMQ, four backend services, a frontend, a base image and a worker image is a lot of moving parts for what a single developer might otherwise do by pasting source into a chat window. If your goal is one-off review of a small contract, the compose stack is overhead you will feel. If your goal is comparing two models or two prompts on the same corpus under identical conditions, the overhead buys reproducibility.

How evmbench differs from running Slither or a hosted auditor

The base image contains slither, which invites the comparison. Slither is a static analyser: it parses Solidity and applies a fixed set of detectors, producing deterministic findings with no model, no API key and no queue. evmbench wraps a language model and asks it to reason about the code, which means results vary with the model, the prompt in backend/worker_runner/detect.md and the model map in backend/worker_runner/model_map.json. Run Slither twice on the same source and you get the same output; run an LLM agent twice and you may not. That variance is the point of a benchmark and the reason evmbench exists, but it is also why you would not use it as a gate in CI.

Against a hosted auditing service, the difference is where the code goes. A hosted service runs your source on someone else's infrastructure under their terms. evmbench runs it in a container you start, on a machine you control, with the caveat that the worker is documented as untrusted. The pinned frontier-evals submodule is the third comparison point: that is the upstream detect evaluation itself, and evmbench is described as a companion interface to it. If you only want to reproduce the published numbers, the submodule is the closer artifact; the UI and job queue here exist to make repeated runs easier to operate.

Licence, maintenance and the cost of upgrading

evmbench is Apache-2.0, and the LICENSE file sits at the repository root. Apache-2.0 is permissive and includes an explicit patent grant, which matters if you intend to fold the harness into internal tooling. The pinned frontier-evals submodule is a separate upstream project with its own licence, and the README does not state what that is; check it before redistributing anything that includes the submodule. Nothing here is legal advice.

There are no retrieved releases, so upgrades mean tracking main. The last push was on 2026-06-24, roughly three months before this writing, and the repository is not archived. That is recent enough that the code is moving, but with no tags you have no version boundary to test against. The Makefile gives you the checks the maintainers run: backend-lint runs ruff via uv with a locked sync, backend-typecheck compiles api/app.py, docker/worker/init.py and the two instancer backends, and docker-build-images rebuilds all four images. Running those three targets before and after a pull is the cheapest way to notice drift. Expect to rebuild the base and worker images on most upgrades, since both are built from Dockerfiles in backend/docker/.

Editorial conclusion

Adopt evmbench if you want a reproducible detect-only harness for comparing models or prompts on contract source you control, and you can run Docker and Bun locally. Do not adopt it if you need an audited hosted scanner, a supported SaaS product, or a tool that runs without an OpenAI key: the README documents direct BYOK as the default, and the worker is explicitly described as an untrusted environment. Before committing, verify three things yourself: that the base and worker images build on your machine, that the model keys in backend/worker_runner/model_map.json match the models your account can call, and whether EVM_BENCH_CODEX_TIMEOUT_SECONDS at its 10800-second default fits your job sizes.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. paradigmxyz/evmbench on GitHub
4. Project website
5. README