Audit: an eight-stage vulnerability discovery agent for your own codebase

Why one big find-the-bugs prompt loses to eight narrow agents

The project is a from-scratch reimplementation of the pipeline Cloudflare described in its Project Glasswing post, which tested an Anthropic preview model against the Cloudflare codebase. The blog argument the README repeats is that real vulnerability discovery rarely comes from asking one large model to find bugs. It comes from many narrow agents working on tightly scoped questions, such as command injection in one specific function with a stated trust boundary. It also comes from deliberate disagreement: a second agent on a different model whose job is to disprove the first agent. The third ingredient is a reachability trace as a gate, because a buggy function an attacker cannot reach is noise. The fourth is a feedback loop, so a reachable bug in one place seeds hunts for the same pattern elsewhere. Audit ships that architecture as a runnable agent: prompts, schemas, a state store, and an orchestrator, MIT-licensed, built by the bettercap author.

The eight stages, from Recon to Report

Each stage is one markdown prompt in prompts/ plus one JSON Schema in schemas/, and the orchestrator passes the schema into the system prompt so outputs are shape-stable on the first try. Recon runs on Opus 4.7 by default and maps the repository into narrowly scoped Hunt tasks. Hunt runs on Sonnet 4.6, one attack class per agent, and compiles or runs proof-of-concept code. Validate runs on Opus 4.7 and tries to disprove findings, deliberately on a different model from Hunt. Gapfill re-queues under-covered areas, and Dedupe clusters findings by root cause. Trace is the gate: it proves that attacker-controlled input actually reaches the sink. Feedback turns reachable traces into new Hunt tasks, and Report emits a schema-validated structured report. The two-model split matters: the agent that claims a bug and the agent that attacks the claim do not share a model, which blunts the failure mode where one model confidently agrees with itself.

Installing and running your first scan

The README quickstart uses a Python virtual environment and the Claude CLI for authentication:

python -m venv .venv && source .venv/bin/activate
pip install -e .
claude setup-token
echo "CLAUDE_CODE_OAUTH_TOKEN=<paste>" > .env
audit auth-check
audit run --repo /path/to/target --run-id my-run
audit status --run-id my-run
audit report --run-id my-run --format md > report.md

The auth step has two paths: if you are already logged in through claude login, nothing else is needed, and claude setup-token generates a one-year OAuth token for CI or non-interactive use, which lands in .env as CLAUDE_CODE_OAUTH_TOKEN. audit auth-check verifies the setup before you spend anything. A run is addressed by --run-id, and status tracks its progress while report writes the final markdown report to a file.

Billing: your subscription, not the metered API

By default the agent bills against your Claude.ai subscription through the login session, and it does not call the metered Anthropic API. The README is specific about the mechanism: the on-disk auth module scrubs ANTHROPIC_API_KEY from the environment so the run cannot silently route around the OAuth flow. Auth resolution follows a fixed order. First, an LLM gateway is used when ANTHROPIC_BASE_URL points away from anthropic.com and ANTHROPIC_AUTH_TOKEN is set; the gateway environment is left intact and only ANTHROPIC_API_KEY is scrubbed, because it would otherwise outrank the gateway token. Second, headless subscription OAuth through CLAUDE_CODE_OAUTH_TOKEN, which the README recommends for CI. Third, interactive credentials from ~/.claude/.credentials.json after claude login, recommended for local development. The practical consequence: the cost of a scan lands on your existing plan ceiling rather than on a separate API invoice.

Routing through OpenRouter and other gateways

Anything that exposes the Anthropic Messages API at a URL with a bearer token works, and the README gives OpenRouter as the worked example:

export ANTHROPIC_BASE_URL="https://openrouter.ai/api"
export ANTHROPIC_AUTH_TOKEN="$OPENROUTER_API_KEY"
export ANTHROPIC_API_KEY=""           # must be explicitly empty / unset
export ANTHROPIC_MODEL="anthropic/claude-sonnet-4-6"
audit auth-check                       # confirms "using LLM gateway at https://openrouter.ai/api"

With a gateway you spend OpenRouter credits instead of a subscription and can put non-Anthropic models on stages. The README attaches three caveats. Per-stage overrides in config/stages.yaml use model names such as claude-opus-4-7, while OpenRouter wants slash-prefixed forms, so the YAML needs editing if you want different providers per stage; otherwise ANTHROPIC_MODEL forces every stage onto one model, which defeats the two-model design. Non-Claude models may not produce schema-compliant JSON as reliably, though the validation and repair turn still applies. Tool-use semantics such as Read, Grep, and Glob belong to the Claude Code CLI rather than the model, so they work with any gateway that speaks the Anthropic Messages API. For Amazon Bedrock, Google Vertex, or Microsoft Foundry, Claude Code has first-class environment flags such as CLAUDE_CODE_USE_BEDROCK=1 that outrank everything else.

Cost containment on a real codebase

The README is direct about scale: a production codebase can produce 15 to 50 Hunt tasks and 25 or more findings to validate, which gets expensive at default concurrency. Three flags bound the blast radius:

audit run --repo /path/to/target \
  --max-concurrency 1 \           # one claude subprocess at a time
  --max-recon-tasks 15 \          # cap initial Hunt fanout
  --max-cost-usd 30               # abort cleanly if exceeded

--max-concurrency 1 serializes the Claude subprocesses, --max-recon-tasks 15 caps the initial Hunt fanout, and --max-cost-usd 30 sets an abort budget. The README states the budget guard fires both between and within stages, and describes a per-task check in Hunt that cooperatively aborts rather than running 30 more tasks past the cap. That last detail is the difference between a budget cap that works and one that only checks at stage boundaries.

Live-target reproduction and what to use instead

When the target has a running deployment, the agents can be pointed at it. Hunt then reproduces each finding against the live service instead of compiling a local proof of concept, Validate rejects findings that do not reproduce, and Trace confirms reachability with real HTTP round-trips. That moves evidence from plausible to demonstrated, on a target you own. The honest alternative for everyday work is a static analysis scanner: SAST tools run fast on every commit and cover known patterns, but they match rules rather than run proof-of-concept code, and they have nothing like the Validate stage trying to disprove a finding. The other alternative is a single-prompt LLM review, which is cheaper but collapses exactly the four properties this pipeline is built around: narrow scoping, disagreement, reachability gating, and feedback. On maintenance: the repository is MIT-licensed, the last push was on 2026-06-10, and there are no tagged releases, so installs track the head of the default branch.

Editorial conclusion

Audit fits developers who want deep, adversarial vulnerability review of a codebase they own or are authorized to test, and who already pay for Claude or a compatible gateway. It is the wrong tool for a quick pre-commit lint pass, and the README says nothing about authorization handling, so point it only at repos you control. Verify auth with audit auth-check, cap the first run with --max-cost-usd, and read the Trace stage output before acting on any finding.

Frequently asked questions

Official sources

1. evilsocket/audit on GitHub
2. Issues
3. License: MIT
4. README