# api-relay-audit: one Python file fetched by curl, pinned to a release tag

> A local audit tool that probes an AI API relay for prompt injection, model substitution, error leakage and stream anomalies, distributed as a single standalone script or as a DeepSeek Harness bundle whose six peer dependencies include five pinned to the same release candidate.

**toby-bridges/api-relay-audit** — Local security audit for AI API relays and LLM proxies: detects prompt injection, model substitution, tool-call rewriting, SSE anomalies, error leakage, and Web3 wallet risks.

- Repository: https://github.com/toby-bridges/api-relay-audit
- Website: https://toby-bridges.github.io/api-relay-audit/
- Stars: 865 · Forks: 82
- Language: Python
- License: AGPL-3.0
- Published: 2026-09-10 · Updated: 2026-09-10 · Language: en
- Canonical page: https://hysenlabs.com/projects/toby-bridges-api-relay-audit

## The entire quick start is one file fetched with curl

There is no install step because there is nothing to install. The documented path is:

```bash
AUDIT_SCRIPT_REF=v2.4.0
curl -fsSL "https://raw.githubusercontent.com/toby-bridges/api-relay-audit/${AUDIT_SCRIPT_REF}/audit.py" -o audit.py

python audit.py --key <YOUR_KEY> --url <BASE_URL> --output report.md
```

So the reference is pinned to a tag first and the script is pulled from `raw.githubusercontent.com` at that tag, which means the audit you run is the audit that was tagged rather than whatever is on the branch. The instructions are explicit about the exception: use `master` as `AUDIT_SCRIPT_REF` only when intentionally testing unreleased changes. The Web3 variant is the same command with one more flag, `--profile web3`. The script's own dependency claim is narrow: it uses Python's standard library plus `curl`, and your API key is sent only to the relay URL you choose.

## Five of the six peer dependencies are the same release candidate

The DeepSeek Harness bundle carries this peer dependency set:

```json
"peerDependencies": {
    "@deepseek-ai/cordis": "^4.0.1",
    "@deepseek-ai/dsh-commands": "0.1.0-rc.6",
    "@deepseek-ai/dsh-credentials": "0.1.0-rc.6",
    "@deepseek-ai/dsh-llm": "0.1.0-rc.6",
    "@deepseek-ai/dsh-settings": "0.1.0-rc.6",
    "@deepseek-ai/dsh-subprocess": "0.1.0-rc.6"
},
```

Five of the six are pinned to the identical version, `0.1.0-rc.6`, an exact release candidate with no caret, while `cordis` is the only one on a range. Pinning five packages to one prerelease build is defensible when they ship as a set, and it is also a hard floor: the bundle does not work with any other build of the DSH command layer. The Node requirement is a range with a gap in it, `^22.19.0 || >=24.0.0`, which excludes Node 23 and anything below 22.19 in the 22 line. The test command runs the Node test runner directly against a glob, `node --test dsh/test/*.test.js`, so the bundle has its own JavaScript tests separate from the Python ones.

## Two distribution modes with different dependency stories

The repository states both modes and they are not the same product. The first is `audit.py`, described as a zero-dependency standalone script for quick local audits, which is the file the quick start downloads. The second is `api_relay_audit/` plus `scripts/`, described as the modular development version with tests. The dependency difference between them is small but real. The modular path is the one with a requirements file, and that file contains a single entry:

```text
httpx>=0.24.0
```

So the standalone claim of standard library plus curl holds for the standalone script, and the modular version adds exactly one HTTP client with a floor and no ceiling. Three runtime profiles exist across both modes: `general` for default AI API relay and LLM proxy checks, `web3` for wallet-safety probes aimed at Web3 agent flows, and `full` for general plus Web3. The Web3 probes are profile-gated, and the documentation is careful to say that general relay audits do not imply wallet safety.

## Four query families and an explicit request not to flatten them

The most unusual thing in this README is a table of boundaries and an instruction to maintain them. Four query families are defined, each with a user intent, the profiles and step numbers it covers, and an evidence boundary. The general API relay audit uses the `general` profile by default and `full` for every probe, and produces a local report rather than a safety certificate. The prompt injection audit runs `general` and covers steps 3 to 6, recording prompt evidence without publishing private prompts or secrets. Model substitution signals covers steps 5, 10, 13 and 14 and is explicit that self-identification, latency and channel fingerprints are signals rather than standalone proof of provider substitution. The Web3 relay audit is step 11 and profile-gated. The contract for this lives in `docs/query-families.md`, and the documentation asks that README headings, Pages cards, issue templates and skill descriptions preserve these boundaries instead of flattening them into one slogan.

## The tool refuses to certify anything and keeps inconclusive probes

Three sentences in the What It Does Not Claim section do more work than the feature list. It does not certify that a relay is safe. It does not replace manual security review or operational monitoring. And it does not treat `inconclusive` as `clean`, because blocked probes and ambiguous responses stay visible in the report. That last one is the design decision that matters most for a tool in this category. A scan that reports a pass count invites the reading that everything unflagged was checked and found fine, and this one refuses that reading by keeping its unknowns in the output. The graded findings are `LOW`, `MEDIUM` and `HIGH` summaries with per-step evidence attached, and the coverage behind them spans prompt safety, relay integrity, model identity and Web3 wallet safety.

## The credential goes through an environment variable and not the command line

The DeepSeek Harness integration makes a point of how the key travels. The command reuses the current DSH provider's `baseURL`, model and credential reference, and the credential stays in DSH Credentials, delivered to the local audit process through an environment variable, never through command arguments or the session log. That is a specific design choice rather than a general one, because the standalone path does the opposite: `python audit.py --key <YOUR_KEY>` puts the key directly on the command line, where it is visible in the process list and in shell history. The plugin-side commands are given as a list:

```text
/relay-audit
/relay-audit --connectivity
/relay-audit --profile web3 --fast-context
```

Passing no arguments preserves the existing full-audit default and may consume metered tokens, and `--connectivity` is the lower-cost check. The bundle also declares its compatibility limits: it adds no new model baseline, the selected route must identify as Claude even though the relay API itself may be Anthropic-compatible or OpenAI-compatible, and independent wrappers without DSH profiles and the DSH command registry are not compatible.

## A Python auditor published as a Node package that re-exports its own script

The bundle is a package.json whose name is `dsh-api-relay-audit` and whose main is a JavaScript entry point, but whose export surface includes the Python file:

```json
"exports": {
    ".": "./dsh/index.js",
    "./audit.py": "./audit.py",
    "./cordis.patch.yml": "./dsh/cordis.patch.yml"
},
```

The packaged file list is short and deliberate: `audit.py`, `dsh/index.js`, `dsh/cordis.patch.yml`, `README.md`, `LICENSE`, `NOTICE` and `VERSION`. So the Python entry point is a first-class export of a Node package, and the audit script travels inside it rather than being fetched separately once installed through the harness. `"private": true` is set, which means this manifest is never published to a registry; installation goes through the harness, which takes a git reference: `dsh plugin --profile web add "github:toby-bridges/api-relay-audit#${DSH_PLUGIN_REF}"`. The licence field reads AGPL-3.0-only, and the manifest carries a `dsh.bundle.patch` pointing at the same cordis patch file it exports.

## Two tags on one day, a VERSION file, and an odd name at the root

Three releases are on record and two of them share a date. v2.3.0 came on 2026-06-07, then v2.3.1 and v2.4.0 both on 2026-08-15, minutes apart. The package manifest version is 2.4.0 and the repository root also carries a plain `VERSION` file, which is included in the packaged file list, so the version exists in at least two places for anyone installing through the harness. The root listing is unusually broad for a tool of this size: `.github/`, `CITATION.cff`, `CODE_OF_CONDUCT.md`, `CONTRIBUTING.md`, `CONTRIBUTORS.md`, `LICENSE`, `NOTICE`, `ROADMAP.md`, `SECURITY.md`, `SKILL.md`, `VERSION`, `assets/`, `audit.py`, `deploy/`, `docs/`, `dsh/`, `requirements.txt`, `scripts/`, `skills/`, `tests/` and `web/`. One of those names does not fit the pattern: `FOR_JOHN.md`, which is the only entry in the repository that is addressed to a person rather than to a function.

## Conclusion

api-relay-audit suits someone about to send real traffic through a third-party relay, mirror, gateway or resale endpoint who wants a repeatable local report instead of a web form that asks for their API key. It does not suit anyone looking for a clean bill of health, because the tool's own documentation says it does not certify that a relay is safe and keeps inconclusive probes in the output. Check four things first. Check the pinned reference you fetch, since the instructions default to a release tag and say to use master only when you intend to test unreleased changes. Check which runtime profile you need, because the Web3 probes are gated behind a profile and a general audit does not imply wallet safety. Check whether the target route identifies as Claude if you are using the DeepSeek Harness bundle, since the distribution adds no new model baseline. And check whether you want the standalone script or the modular version, since one has no dependencies and the other needs httpx. The licence is AGPL-3.0-only, the default branch is master, and it was last pushed on 2026-09-16.

## FAQ

### What does api-relay-audit check for?

It probes an AI API relay or LLM proxy for prompt and context signals such as hidden injection, instruction override and context truncation, for response integrity problems such as error leakage and SSE stream anomalies, and for model identity signals. Web3 wallet probes cover transfer guidance, signed-transaction refusal and private-key refusal.

### Does api-relay-audit guarantee a relay is safe?

No, and it says so directly. It does not certify that a relay is safe, it does not replace manual security review or operational monitoring, and it does not treat an inconclusive result as clean, so blocked probes and ambiguous responses stay visible in the report.

### How do I install and run api-relay-audit?

Fetch the single script at a pinned reference with curl, then run it with your key, your relay URL and an output path. The reference defaults to the v2.4.0 tag, and the documentation says to use master only when intentionally testing unreleased changes.

### What are the runtime profiles in api-relay-audit?

Three: general for default AI API relay and LLM proxy checks, web3 for wallet-safety probes aimed at Web3 agent flows, and full for general plus Web3. The Web3 checks are profile-gated, and a general relay audit does not imply wallet safety.

### Can api-relay-audit run inside the DeepSeek Harness?

Yes, as an installable bundle added with a pinned git reference through the dsh plugin command on a profile. Five of its six peer dependencies are pinned to the same release candidate, 0.1.0-rc.6, and the bundle requires a Node version matching ^22.19.0 or 24 and newer.

## Sources

- [License: AGPL-3.0](https://github.com/toby-bridges/api-relay-audit/blob/master/LICENSE)
- [Project website](https://toby-bridges.github.io/api-relay-audit/)
- [README](https://github.com/toby-bridges/api-relay-audit/blob/master/README.md)
- [Releases](https://github.com/toby-bridges/api-relay-audit/releases)
- [toby-bridges/api-relay-audit on GitHub](https://github.com/toby-bridges/api-relay-audit)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/toby-bridges-api-relay-audit