Veridrop: Checking Whether an AI API Relay Serves the Model It Claims

The relay trust problem Veridrop was built to answer

A relay sits between your code and Anthropic, OpenAI or Google. You send an OpenAI-shaped request to a base_url you do not control, and a response comes back that looks correct. Nothing in that exchange proves which model produced the tokens. The relay can route to a cheaper model, drop capabilities it cannot pass through, or rewrite usage numbers, and the client sees a well-formed response either way.

Veridrop is built for that gap. The README frames it as an authenticity and compliance checker for relay and proxy services, and the input is deliberately minimal: a base_url, an api_key and a model name. It then answers three questions, in the project's own wording: whether the relay forwards to the model it claims, whether advanced features such as PDF, tool use, thinking, function calling and long context are silently stripped, and whether response fields, streaming event sequences and usage accounting match the official specs.

The intended user is a developer or a team buying API capacity through a middleman, plus anyone maintaining a relay and wanting an outside check. It is not aimed at people calling api.anthropic.com directly, where the answer to 'is this really Claude' is not in question. The repository also ships a hosted instance at veridrop.org that the README describes as free, signup-free and non-persistent of API keys, so the tool can be evaluated without cloning anything.

How the probes work: detectors, baselines and a Bayesian leaderboard

The mechanism is a suite of probe requests, not a single call. According to the README, each protocol is covered by a set of named detectors grouped under three categories: authenticity, capability and protocol. Claude gets twelve detectors, OpenAI Chat Completions gets eight, and the Gemini OpenAI-compatible API gets seven. Each detector targets a specific observable, from asking the model who it is, to checking that tool call IDs carry the expected prefix, to comparing streaming and non-streaming responses for the same request.

The results are then diffed against official-API baselines. The repository keeps these under data/baselines/, and the CLI has a compare subcommand that looks them up automatically. That diff is what turns a pile of probe results into a score and a list of differences.

The design has one clear asymmetry. Under the Claude protocol, a thinking-enabled response carries a signature field that the README describes as roughly 500 to 2000 characters of server-side cryptographic output, weighted at 25 percent of the Claude protocol score because a relay cannot forge it. The README states this is the only cryptographically verifiable indicator in the tool. OpenAI and Gemini have no equivalent, so verification there stops at the protocol and behaviour level. The README still claims a useful trick for those: leftover backend fingerprints in the usage object, such as claude_cache_creation_* fields, usage_source: anthropic, or Anthropic naming like input_tokens and output_tokens appearing in an OpenAI-shaped response. When those appear, the README says the verdict is capped at marginal.

The public leaderboard is described as Bayesian-weighted and aggregated from community detections. The README gives running totals, and it also states that commercial cooperation does not change detection scores, verdicts, critical issues or the leaderboard algorithm.

Installing Veridrop and running a first detection

Self-hosting starts from the repository. The README gives a clone, a virtual environment and an editable install with the dev and web extras, which pulls in FastAPI, uvicorn, Jinja2, Pillow and python-multipart on top of the core httpx, pydantic, typer, rich and rapidfuzz dependencies. Python 3.10 or newer is required.

git clone git@github.com:canarybyte/veridrop.git
cd veridrop
python3 -m venv venv
./venv/bin/pip install -e ".[dev,web]"

Credentials come from a .env file copied from the example. The example defines three keys, and the README points at the Anthropic set for the CLI walkthrough. Point ANTHROPIC_BASE_URL at the relay you want to test, not at the official endpoint, if the goal is to check a relay.

cp .env.example .env
nano .env

A ping is the cheap first step. The README calls it a connectivity test that takes seconds and costs almost nothing, and it takes a model argument.

./venv/bin/relay-detector ping --model claude-haiku-4-5

If that returns, run the full suite. The README quotes roughly one minute and about $0.012 for this example, and writes the report to a JSON file.

./venv/bin/relay-detector detect \
  --model claude-haiku-4-5 \
  --mode full \
  -o out/test.json
./venv/bin/relay-detector compare out/test.json

The compare step prints a baseline-versus-relay table. In the README's sample output a relay scores 63.1 against a baseline of 100.0, with the thinking signature check at zero. The same console entry point is installed under two names, relay-detector and veridrop.

Long context verification and what it costs you

The long_context probe is the part that catches claims a normal request would never expose. The README describes three layers of needle-in-haystack probes, and the pitch is specific: a relay advertising a 1M context window while actually serving 200k, or forwarding to a small-window model, or failing the middle of a long prompt. The README states that basic protocol detection cannot see any of this.

There are two tiers. The standard tier probes at 32k, 100k and 200k tokens. The extreme tier scales to the model's stated ceiling, so a 1M model is probed at 32k, 500k and 950k. The README says the run stops at the first layer that fails, which avoids burning tokens on the remaining layers.

The cost is the constraint worth reading twice. These probes are paid for with your own API key. The README puts the standard tier at roughly $0.05 to $0.50 and the extreme tier at roughly $0.05 to $8, depending on the model's ceiling. Long context is opt-in rather than part of the default run, and it covers Claude and OpenAI only; the README states Gemini is not supported for this probe. That is a real gap for anyone whose relay serves Gemini, because the context window is exactly where an inflated claim is easiest to hide.

Where Veridrop gives you a weak answer

The honest reading of the README is that Veridrop's confidence is uneven across the three protocols, and the project says so. Cryptographic verification exists only for Claude. For OpenAI and Gemini, the strongest stated evidence is protocol-level and behavioural, plus the backend-fingerprint check on the usage object. A relay that is careful about field shapes and does not leak Anthropic naming into an OpenAI response would not trip that detector, and the README does not describe a second line of defence at the same strength.

There is also a scope limit in what the tool measures. It checks authenticity, capability and protocol compliance. It does not check uptime, latency, rate-limit behaviour under load, or whether a relay logs your prompts. A relay can pass every detector and still be a poor operational choice, and the README makes no claim otherwise.

The third limit is cost and key exposure. Every probe consumes tokens billed to the key you supply. The README's own figures for a full Claude run and for the extreme long-context tier mean that testing many relays, or re-testing on a schedule, is a real spend. And you are handing a third-party relay, or the hosted service, a working key for the duration of the run. The README states the hosted service never persists keys; it does not make the same claim about the relay you are testing, because that is outside its control.

Veridrop against rolling your own probe script

The obvious alternative is a script you write yourself. A few dozen lines with httpx can send a request to a relay and print the response. The difference is in what you compare against. A homegrown script usually checks that the call succeeded and that the text looks plausible, which is exactly the check a swapped model passes.

Veridrop's value is the baseline corpus and the detector set. The repository ships data/baselines/ for the compare step, and the Claude protocol alone carries twelve named detectors covering identity, behavioural fingerprints, the thinking signature, response consistency, company knowledge, PDF extraction, tool_use schema, protocol fields and SSE sequence, stream versus non-stream integrity, message ID prefixes, usage accounting and long context. Rebuilding that, and keeping it current as providers change their response shapes, is the work you are avoiding.

A second alternative is a proxy or gateway that sits in front of the relay and logs traffic. That gives you observability over your own requests but no verdict, because it has no official baseline to diff against and no way to test a signature. If you already run such a gateway, Veridrop is complementary: it tells you whether the upstream is what it claims, and the gateway tells you what your own traffic looks like.

A third option is simply using the hosted service. For a one-off check that is faster than installing anything, and the README positions it as the no-clone path. Self-hosting matters when you want the detection logic, scoring weights and field evidence in your own environment, which the README explicitly lists as the reason the full logic lives in the repository.

Licence, maintenance and the cost of keeping up

Veridrop is licensed AGPL-3.0, and pyproject.toml declares AGPL-3.0-or-later with the LICENSE file listed. The practical consequence for most readers is that if you modify the code and run it as a network service that others interact with, the AGPL's source-availability condition applies to your modified version. Running it internally to check your own relay is the ordinary case and does not raise that question. This is a description of the licence text, not legal advice; if you plan to build a commercial service on top of the code, have someone qualified read the AGPL.

On maintenance, the repository is not archived, and the last push was on 2026-08-15, the same day as the v0.1.0 release. That is a single release at version 0.1.0, so treat the API surface, the detector weights and the report format as young. The upgrade cost is not in the Python dependencies, which are ordinary and loosely pinned. It is in the baselines. A tool whose verdicts come from diffing against official-API baselines needs those baselines updated when providers change response shapes, and a version 0.1.0 project has not yet demonstrated a cadence for that. The repository does carry DESIGN.md, CHANGELOG.md, CONTRIBUTING.md, SECURITY.md and OPENAI_API_PROTOCOL_VALIDATION.md, which is a reasonable documentation footprint for a first release, and DESIGN.md is where the README points for weight allocation and sub-check detail. If you pin to v0.1.0 and the baselines drift, your scores drift with them, so a re-run after an upgrade is the only way to know whether a score change came from the relay or from the tool.

Editorial conclusion

Adopt Veridrop if you buy Claude, OpenAI or Gemini access through a relay and want field-level evidence rather than a vendor's word, and if you are willing to spend your own API key on the probes. Do not adopt it as a general proxy monitor, a latency or uptime tool, or as a way to audit a Gemini relay's context window, since long_context covers Claude and OpenAI only. Before trusting a verdict, run relay-detector ping to confirm connectivity, then detect --mode full and compare against data/baselines/, and read DESIGN.md for the weightings behind the score.

Frequently asked questions

Official sources

1. canarybyte/veridrop on GitHub
2. License: AGPL-3.0
3. Project website
4. README
5. Releases