RefChecker validates citations against five databases and an LLM web search

One engine, three front ends: web UI, CLI, HTTP API

RefChecker validates reference accuracy in academic papers, and the stated goal is to catch citation errors, fabricated references and metadata mismatches before they reach reviewers. The verification sources are named up front: Semantic Scholar, OpenAlex, CrossRef, DBLP and the ACL Anthology. Alongside them, the tool uses LLM-powered deep web search to flag references that look fabricated.

The deployment surface is what makes it usable in different settings. The same engine is exposed through a web UI, which is also packaged as a native desktop application, a CLI for single papers and scripted runs, and an HTTP API described in docs/web-ui.md. A feature matrix in the README maps the capabilities across web, desktop, CLI and API, so an institution can put the web UI in front of students and keep the CLI for a checking pipeline.

The desktop builds are Tauri based, with a macOS arm64 dmg, a Windows x64 msi, and Linux AppImage and deb packages, built and signed by GitHub Actions on every release tag. The project was built by Mark Russinovich with AI assistants, and a deep dive video is linked from the README.

Quick start: pip for a single paper, Docker for the web UI

The quickest route on a laptop is pip. The CLI installs with the llm extra, which pulls the OpenAI, Anthropic and Google clients, and then takes either an arXiv identifier or a path to a PDF:

pip install academic-refchecker[llm]
academic-refchecker --paper 1706.03762
academic-refchecker --paper /path/to/paper.pdf

The web UI has its own extra, which adds the FastAPI and uvicorn stack, and a launcher command:

pip install academic-refchecker[llm,webui]
refchecker-webui

For a shared instance, the container image publishes port 8000:

docker run -p 8000:8000 ghcr.io/markrussinovich/refchecker:latest

After that the interface is at http://localhost:8000 in a browser.

One performance note from the README matters for a first run: setting SEMANTIC_SCHOLAR_API_KEY brings verification to 1 to 2 seconds per reference against 5 to 10 seconds without it, which is the difference between a tolerable check and an annoying one on a fifty reference paper.

Hallucination detection is a two stage decision, not a flag

The part that separates RefChecker from a metadata differ is the hallucination path, and the interesting detail is that it does not fire on the first database match. The flow is: deterministic pre-filters, then LLM deep web search, then metadata re-verification when the LLM finds a better match. Only after the citation is checked against the metadata of the LLM-found source does RefChecker decide whether it is an error or a hallucination.

That ordering is the whole design. A reference that misses in every database is not automatically fabricated, and a reference that matches but with a different author list is not automatically a hallucination either. The deep search step is what looks for the real paper behind a plausible-looking citation, and the re-verification step is what prevents the tool from trusting an LLM search result at face value.

Deep hallucination checks also require a provider capable of them: OpenAI, ChatGPT, Anthropic, Google or Azure. Without such a provider configured, the deterministic layer still does its work, but the part of the feature that looks for the real source cannot run.

Input formats, extracted fields, and the GROBID fallback

The tool accepts arXiv IDs and URLs, PDFs, LaTeX .tex files, BibTeX .bib and .bbl files, and plain text, which covers the ways a bibliography reaches a reviewer. Once parsed, the metadata checks cover titles, authors, years, venues, DOIs, arXiv IDs and URLs, and the matching layer is designed to tolerate formatting variation, with the README giving BERT against B-ERT and pre-trained against pretrained as the examples of differences that should not count as errors.

Extraction is the weak point, and the README is honest about it. LLM extraction is generally more accurate, but PDFs can fall back to GROBID when no extraction LLM is configured, which means a PDF-only run without a model configured gets machine extraction rather than model extraction. That distinction changes the error profile: a mis-parsed title shows up as a metadata mismatch, and a report full of mismatches may be a parsing artefact rather than a bibliography full of mistakes.

The dependency list reflects the same range, with pypdf, pdfplumber and pymupdf for documents, arxiv for the identifier path, pybtex for bibliography parsing, and fuzzywuzzy with python-Levenshtein for the tolerant string comparison.

Bulk scanning and OpenReview venue audits

The scale at which this is used is where it gets interesting. Bulk checking takes multiple files or a ZIP in the web UI, and the CLI has --paper-list and --openreview for scripted runs. The OpenReview integration goes further and fetches all accepted, or all submitted, papers for a venue and scans them in one command.

The README describes RefChecker as the pipeline behind a paper titled Phantom References: Hallucinated Citations That Survive Peer Review at Top-Tier Conferences, which used the bulk scan to audit every accepted paper at ICLR, ICML, NeurIPS and USENIX Security. The figures given are that in 2025 roughly one in twenty NeurIPS and USENIX Security papers carried at least two likely hallucinated references, at about $0.04 per paper. Those are the paper's numbers as reported in the README, and the rate is a claim about a specific corpus rather than a general expectation for any venue.

For a reviewer or an author, the practical value is a bibliography check before submission. For a programme chair, it is a venue-wide scan that would be impossible to do by hand.

AI-generated-text detection is opt-in and explicitly advisory

One feature in the table deserves scepticism, and the README supplies the caveat itself. AI-generated-text detection is opt-in and analyses the body text of each checked article for AI-generated-likelihood, returning a low, medium or high band together with advisory flagged passages. There are three engines: a local calibrated model that works offline and is downloadable, an LLM judge that reuses the configured LLM, and an external API such as Pangram or GPTZero.

The README is explicit that the result is advisory only, that detection is unreliable on technical and non-native-English academic writing, and that results are framed as a self-check and never as proof of misconduct. That framing is the correct one, and it is also the reason the feature sits behind a setting rather than in the default path.

There is a practical cost as well. The local model pulls in nltk and scikit-learn, and the requirements file notes that the local backend is deliberately not bundled into the desktop sidecar to keep the desktop build small, so a local model has to be installed separately.

Container configuration, multi-user mode, and an optional second tier

The Docker image is built in three stages, building the frontend with node, then the Python dependencies in a virtual environment, then a runtime image, and the compose file is the quickest way to see the configuration surface. The environment block passes the LLM keys through, and the ones that matter are ANTHROPIC_API_KEY, OPENAI_API_KEY and GOOGLE_API_KEY, along with SEMANTIC_SCHOLAR_API_KEY and PAPERCLIP_API_KEY for verification, REFCHECKER_DATA_DIR for storage, and GROBID_URL pointing at the bundled GROBID service.

Multi-user mode is switched on with REFCHECKER_MULTIUSER, with JWT_SECRET_KEY for token signing, SITE_URL and HTTPS_ONLY for the deployment, REFCHECKER_ADMINS for the admin list, and MAX_CHECKS_PER_USER, which defaults to 3, to cap how many checks one account can run. OAuth for Google and GitHub is configured through client id, secret and redirect URI variables, and a Render deployment is covered by render.yaml.

One more tier is worth knowing about. The requirements list a Paperclip biomedical full-text and arXiv search used as an optional secondary verification tier, with the checker at src/refchecker/checkers/paperclip.py described as inert when PAPERCLIP_API_KEY is missing, so it is safe to ship by default.

Beta packaging, permissive licence, and the limits of the verdict

The project metadata describes itself as Beta, and the licence is MIT with the text in the LICENSE file. The releases move fast: v3.1.4, v3.1.5 and v3.1.6 all landed on 2026-10-11 and 2026-10-12, and the last push to the repository was on 2026-10-12. Release highlights are moved into docs/RELEASE-NOTES.md, with the full list on the releases page.

The Python requirements are worth noting for deployment. The package requires Python 3.11 or newer and its classifiers cover 3.11, 3.12 and 3.13, and the dependency pins are aggressive, with recent floors across requests, pandas, numpy, pypdf and pdfplumber, so a locked environment in an institution may need refreshing before an upgrade.

Where RefChecker is the wrong tool is straightforward. It does not judge whether an argument is sound, only whether its citations point at work that exists, and it reports metadata mismatches and likely fabrications as separate categories with different confidence. A verdict of hallucination is a re-verified search result, not proof, and the corrections it can generate, in BibTeX, plain text and bibitem form, are the part a careful author should read line by line before accepting.

Editorial conclusion

RefChecker fits authors, reviewers and venue organisers who need a defensible check of a bibliography before submission, and the OpenReview scanning path is its strongest feature for large venues. It does not fit as a one-click quality gate, because deep hallucination checks need a capable LLM provider, PDF parsing falls back to GROBID when no extraction model is configured, and AI-generated-text detection is explicitly advisory. Before you rely on it, check three things: that the metadata for each error is a mismatch you can correct rather than a real second reference with a formatting difference, that a hallucination verdict rests on the re-verified LLM match rather than on the first database hit, and that your Python is 3.11 or newer, which the package requires. The code is MIT licensed, the classifier still says Beta, and the last push was on 2026-10-12.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. markrussinovich/refchecker on GitHub
4. README
5. Releases