API Relay Audit: a local probe suite for third-party LLM relays
Local security audit for AI API relays and LLM proxies: detects prompt injection, model substitution, tool-call rewriting, SSE anomalies, error leakage, and Web3 wallet risks.
At a glance
- What is it?
- API Relay Audit is a Python script that sends structured probes through an OpenAI- or Claude-compatible relay and writes a Markdown report with a LOW, MEDIUM or HIGH verdict. It is useful for pre-adoption checks on a relay you do not control, and it explicitly refuses to certify anything.
- Who is it for?
- Adopt it if you route production or wallet-adjacent traffic through a relay you did not build and you want a repeatable, local, reviewable artifact before you commit. Do not adopt it as a gate that returns pass or fail, and do not read a clean general run as wallet safety.
- Can I use it commercially?
- Yes, with strict conditions. AGPL-3.0 is a network copyleft licence: if people use a modified version over a network, for example as a hosted service, you must offer them its source code under the same licence.
- Is it still maintained?
- Yes. The repository last received commits 3 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The relay trust problem this tool is aimed at
When you call a model through a third-party relay, mirror or resale API, the wire between your client and the model is not yours. The endpoint can rewrite the system prompt, drop earlier turns, answer with a different model than the one you asked for, or alter a tool call before your agent executes it. None of that is visible in a normal 200 response. The README frames the tool for exactly this situation: you rely on "a third-party AI API relay, OpenAI-compatible proxy, Claude-compatible proxy, or Web3 agent workflow" and want a repeatable report before sending real traffic.
The intended user is narrow. This is not a general LLM observability tool and not a runtime guardrail. It is a pre-adoption probe suite you run by hand, one relay at a time, and read. The repository's own framing is defensive: the README states plainly that it "does not certify that a relay is safe" and does not replace manual review or operational monitoring. That honesty is the most useful thing about the project's positioning, because a scanner that outputs a verdict invites people to treat the verdict as a decision.
How the probes are organised: separate query families
The design choice that distinguishes this project from a single pass-or-fail scanner is the split into four query families, each with its own declared evidence boundary. The README table lists them as API relay audit, prompt injection audit, model substitution signals, and Web3 relay audit. Each family maps to a profile and a set of steps. Prompt injection work sits in Steps 3 through 6. Model identity signals are collected in Steps 5, 10, 13 and 14. The Web3 wallet probes are Step 11 and are profile-gated.
That gating has a consequence the documentation is explicit about: a general relay audit does not imply wallet safety. If you run the default profile and see a clean report, the wallet-sensitive probes never executed. The README also states that self-identification, latency and channel fingerprints are signals rather than standalone proof of provider substitution. So a report can show a model identity anomaly that is not, on its own, evidence that the relay swapped models. The canonical contract for these boundaries lives in docs/query-families.md, and the README asks that headings, Pages cards, issue templates and skill descriptions preserve them rather than flattening them into one claim. Whether downstream consumers respect that is not something the repository can enforce.
Running the standalone audit.py against a relay
The zero-dependency path is a single file fetched from a pinned tag. The README gives these commands:
AUDIT_SCRIPT_REF=v2.4.0 curl -fsSL "https://raw.githubusercontent.com/toby-bridges/api-relay-audit/${AUDIT_SCRIPT_REF}/audit.py" -o audit.py python audit.py --key <YOUR_KEY> --url <BASE_URL> --output report.md
For wallet-sensitive work, add the profile flag: python audit.py --key <YOUR_KEY> --url <BASE_URL> --profile web3 --output report.md. The documented runtime profiles are general (the default), web3, and full, where full is general plus the Web3 checks. The README states the standalone script uses only the Python standard library plus curl, and that your API key is sent only to the relay URL you choose. That last point is the practical reason to prefer this over a hosted scanner that asks for a key.
Pinning matters here. The README warns to use master as AUDIT_SCRIPT_REF only when intentionally testing unreleased changes, and the DSH plugin instructions repeat the same discipline with DSH_PLUGIN_REF. Fetching audit.py from a moving branch means the code you execute can change between runs, which undermines the point of keeping a repeatable report. The output is a structured Markdown report with per-step findings and a final LOW, MEDIUM or HIGH verdict. The repository points to a sanitized fixture at docs/examples/sanitized-audit-report.md if you want to see the shape before running anything.
The DSH plugin path and where your credential goes
Besides the standalone script, the repository is an installable dsh-api-relay-audit bundle for DeepSeek Harness surfaces that use the @deepseek-ai/dsh-commands registry. Installation is pinned to a release tag:
DSH_PLUGIN_REF=v2.4.0 dsh plugin --profile web add "github:toby-bridges/api-relay-audit#${DSH_PLUGIN_REF}"
A second form targets profile-based clients such as dsh-cc-tui: dsh plugin --profile cc-tui add "github:toby-bridges/api-relay-audit#${DSH_PLUGIN_REF}".
The plugin reuses the current DSH provider's baseURL, model and credential reference. According to the README, the credential stays in DSH Credentials and reaches the local audit process through an environment variable, never through command arguments or the session log. That is the right split: command arguments and session logs are the two places a key leaks most easily. The documented commands are /relay-audit, /relay-audit --connectivity, /relay-audit --profile web3 --fast-context, and a fully specified form with --url, --model and --credential-ref.
One warning is worth repeating because it affects cost, not just security. The README states that running /relay-audit with no arguments preserves the existing full-audit default and may consume metered tokens, and that --connectivity is the lower-cost check. If you are auditing a relay billed per token, an unpinned habit of running the bare command is an expense line. The README also notes that this distribution does not add a new model baseline, so the selected route must identify itself in a way the checks can work with.
What the coverage list does and does not establish
The README's coverage list names prompt safety checks (token injection, prompt extraction, instruction override, jailbreak resistance), relay integrity checks (context truncation, tool-call substitution, error leakage, stream integrity), model identity checks (non-Claude identity leaks, substitution signals, Claude/OpenAI-compatible relay behaviour), and Web3 wallet safety checks (transfer guidance, signed-transaction refusal, private-key refusal).
Read that as a list of probe categories, not a list of guarantees. Several of these detections are inherently signal-based. Context truncation can be inferred by comparing what you sent with what the model appears to have seen, but a relay that truncates only under load, or only for long conversations, may pass a short probe set. Tool-call rewriting is only observable if the probe exercises tool calls in a way the relay chooses to alter. The README's own caveat about model substitution applies broadly: self-ID, latency and channel fingerprints are signals, not standalone proof. The report format keeps this honest by leaving blocked probes and ambiguous responses visible as inconclusive rather than folding them into clean. A report full of inconclusive rows is a real result, and it means your probe set did not get a usable answer, not that the relay passed.
The failure mode to plan for, and the alternative
The genuine limitation is the one the project states and then works around: the verdict is a summary of what the probes could observe, and a relay that behaves well under audit and differently under production traffic is not detectable by a one-shot run. The tool is also the wrong instrument if you need runtime enforcement. It produces a Markdown file, not a proxy, not a policy engine, and not an alerting path. If your concern is a relay changing behaviour next week, a static report from today does not address it, and the README says as much by declining to replace operational monitoring.
The obvious alternative is to put a logging proxy of your own between the client and the relay and inspect real traffic. That is a different approach with a different cost profile. A local proxy sees every request and response, including the ones you actually care about, and can be left running. It also requires you to operate and store that traffic, and it only shows you what the relay does to your real prompts, not what it does when probed deliberately. API Relay Audit inverts that: it sends constructed probes and keeps nothing but the report. The two are complementary, and neither substitutes for the other. If you already run your own gateway in front of the relay, you have the observation point and the audit script becomes a way to generate specific adversarial cases to look for in your logs.
Licence, maintenance and what to check before relying on a report
The project is AGPL-3.0. That matters if you intend to fold the code into a hosted service or an internal tool that other people reach over a network, because the licence carries network-use obligations. I am not giving legal advice here; if you plan to redistribute it or run it as part of a service, read the licence text and get your own answer. For the common case of downloading audit.py and running it locally against a relay, the licence is unlikely to be the deciding factor.
Maintenance signals from the material: the default branch is master, the repository is not archived, and the last push is 2026-08-29. The release history shows v2.3.0 in June 2026, then v2.3.1 and v2.4.0 on the same day in August 2026, which is consistent with a quick follow-up fix rather than a long release cadence. The README documents two distribution modes, the standalone script and a modular api_relay_audit/ package plus scripts/ with tests, so there is a tested development path alongside the single-file artifact. Upgrade cost is low by design: the script is fetched by tag, the DSH plugin is installed by tag, and neither adds a model baseline. The real recurring cost is token spend on audits, which the README flags for the bare /relay-audit command.
Before you act on any report, confirm which steps your profile executed by reading docs/query-families.md, and check the inconclusive rows rather than the verdict line. Then decide whether the relay is worth a second run under --profile full, and whether your own gateway logs can corroborate what the probes found.
Editorial conclusion
Adopt it if you route production or wallet-adjacent traffic through a relay you did not build and you want a repeatable, local, reviewable artifact before you commit. Do not adopt it as a gate that returns pass or fail, and do not read a clean general run as wallet safety. Before trusting a report, open docs/query-families.md and confirm which steps your profile actually executed, then rerun with --profile full and compare the per-step findings, including the inconclusive ones.
Community notes