hlwy-ai-checker: statistical fingerprinting for third-party AI API resellers
检查第三方AI API是否掺假以及渠道一致|基于llm指纹的AI模型识别
At a glance
- What is it?
- The project asks an LLM to pick random numbers, repeats the request, and compares the output distribution of a reseller channel against a calibration run on the official API. It is a statistical signal generator, not proof, and the README says so itself.
- Who is it for?
- Adopt it if you already buy model access through resellers and want a numeric similarity score to decide whether a channel is worth a second look, and you are willing to run the calibration pass yourself against the official endpoint. Do not adopt it as evidence in a refund dispute or as a compliance control: the README states plainly that results cannot serve as an absolute legal or factual basis for commercial disputes.
- Can I use it commercially?
- Yes, with conditions. LGPL-2.1 is a weak copyleft licence: you can use it inside commercial and closed-source software, but if you distribute changes to its own files, you must publish those changes under the same licence.
- Is it still maintained?
- Yes. The repository last received commits 9 days ago.
- What is it written in?
- Mainly HTML, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The reseller trust problem this tool tries to quantify
Anyone who buys model access through a reseller rather than a first-party endpoint is making an unverifiable assumption: that the model answering the requests is the model named in the price list. The README frames the project around exactly that gap, describing it as a check for whether a third-party AI API is adulterated and whether channels are consistent. The intended user is not a model provider. It is a buyer, integrator or reseller auditor who holds both an official API key and a third-party channel key for the same model and wants a comparison between them. The scope is narrow on purpose. The tool does not measure latency, throughput, uptime or price, and it does not attempt to identify which model is actually behind a channel. It only produces a similarity judgement between two sets of sampled outputs.
Why asking a model to pick random numbers produces a usable signal
The mechanism rests on an argument the README makes directly: large language models are not true random number generators. When a model is asked to randomly choose numbers, its output is shaped by training data, architecture, RLHF alignment, tokenization strategy and sampling parameters, and those influences leave statistical bias in the distribution. Repeat the same prompt many times and the resulting distribution becomes a fingerprint with some discriminating power. The tool exploits that by comparing the distribution from a third-party channel against a distribution collected from the official endpoint. The README is careful about what this means. It calls the method statistical detection and lists the factors that move the result: model version, sampling parameters, server-side configuration, sample size and request success rate. It then states that the method cannot on its own prove that a third-party channel did or did not use a particular model. That sentence is the most important one in the repository, and it sits in the principles section rather than buried in a footnote.
Calibration first, verification second: the two-stage data flow
The workflow is explicitly two-stage, and the order matters. In manual mode, step one is model calibration against an official API key, with the README noting that the Base URL must include /v1. Step two fills in the third-party channel's API key and Base URL, selects the same model used during calibration, and starts the test. Step three is reading the result, which the README describes as a comparison of fingerprint similarity and related statistics between the official and third-party channels. Because the comparison is relative rather than absolute, the calibration run defines the baseline. If you calibrate against a different model, a different version, or different sampling parameters than the channel is serving, the similarity score is measuring your own configuration mismatch as much as the channel. The README also offers an automatic mode built around a one-click test function, which the screenshots suggest is the intended path for users who do not want to manage the two stages by hand. The README does not document what the automatic mode does differently under the hood, so treat it as a convenience wrapper rather than a separate method.
Getting it running: ZIP, start.py, and the /v1 requirement
Distribution is by ZIP from the Releases page rather than by package manager, and the README asks you to extract it completely before running anything. The single documented entry point is python start.py, executed from the extracted directory. There is no documented pip install step, no requirements file mentioned in the README, and no Docker image referenced, so dependency resolution is something you discover by running the command and reading the error. The only configuration keys named in the material are the API key and Base URL fields, entered separately for the official calibration stage and the third-party verification stage, plus a model selector that must match across both stages. The README's one concrete formatting constraint is that the Base URL must include /v1. The stated deployment property is that the whole thing can run locally, which the README presents as protection for your API key and test data. That claim is consistent with a local Python entry point, but the README does not describe any outbound telemetry or third-party service, so the local-only property is asserted rather than demonstrated in the material available.
Token cost, sampling variance and the failure modes the README admits
The README lists low token consumption as a feature, which follows from the design: the probe is repeated short random-number prompts, not long generations, so a fingerprint run costs far less than a task-based evaluation would. The trade-off is statistical. A small sample size makes the distribution estimate noisy, and the README names sample size and request success rate as factors that affect results. A channel that drops or throttles a fraction of requests does not just slow the test down; it biases the sample, because the requests that fail may not be missing at random. Version drift is the other failure mode. If the official endpoint serves a newer model revision than the one the channel proxies, the fingerprints diverge for reasons that have nothing to do with substitution. The README's consistency claim, that the method is less affected by random factors, is a relative statement about this probe design compared with looser behavioural tests, not a guarantee that two runs of the same channel will agree. The wrong tool case is clear from the README's own framing: if you need a definitive answer for a refund claim or a contractual dispute, this is not that instrument.
How this differs from task-based API evaluation
The obvious alternative is a benchmark harness such as lm-evaluation-harness, which scores a model on accuracy across a task suite. The difference in approach is what gets measured. A task harness asks whether the endpoint produces correct answers, which is what you care about operationally but also what a reseller can most easily imitate: a cheaper model with a good system prompt can pass a surprising share of easy tasks. hlwy-ai-checker never checks correctness. It treats the model as a biased random source and compares the shape of that bias, which is harder to fake because it is a property of weights and tokenizer rather than of prompting. The cost is interpretability. A benchmark gives you an accuracy number you can argue about; this tool gives you a similarity score against your own baseline, and the README explicitly declines to treat that score as proof. A second alternative is simply not buying through resellers, which removes the problem instead of measuring it. That is worth stating because for many teams it is the correct answer, and this tool exists for the cases where it is not available.
Licence, maintenance and what a 2.5-pre1 tag implies
The repository is licensed under LGPL-2.1, which is a copyleft licence with specific obligations around modified library versions and relinking. If you plan to modify the checker and redistribute it, or to bundle it into a larger distributed product, read the licence text and get your own advice; nothing here substitutes for that. On maintenance, the material shows a 2.5-pre1 pre-release dated 2026-09-01, a 2.4.0 release on 2026-08-28 and a 2.3.0 release on 2026-08-10, with the last push to the default branch on 2026-09-06. That cadence suggests active work, and the pre-release tag suggests the project ships incrementally rather than holding changes for a stable cut. The practical upgrade cost is low in one sense and non-zero in another. There is no package manager involved, so upgrading means downloading a new ZIP and extracting it, and there is no documented migration or config-format stability commitment. Because the calibration baseline is tied to the tool's probe behaviour, a version bump that changes the probe prompts could invalidate fingerprints you collected earlier, and the README does not describe a fingerprint format or a compatibility policy. Re-run calibration after any upgrade rather than comparing new channel results against old baselines.
Editorial conclusion
Adopt it if you already buy model access through resellers and want a numeric similarity score to decide whether a channel is worth a second look, and you are willing to run the calibration pass yourself against the official endpoint. Do not adopt it as evidence in a refund dispute or as a compliance control: the README states plainly that results cannot serve as an absolute legal or factual basis for commercial disputes. Before trusting a verdict, verify three things on your own setup: that your Base URL includes /v1 as the calibration step requires, that your official and third-party runs use the same model name and sampling parameters, and that your request success rate is high enough that the sample count you paid for is the sample count you actually got.
Community notes