MetaScreener: multi-LLM ensemble screening for systematic reviews

The screening bottleneck MetaScreener targets

Systematic review screening is a two-pass filter. In the first pass a reviewer reads titles and abstracts from a search export, often thousands of records, and decides include or exclude against pre-registered criteria. In the second pass the same reviewer reads the full PDF of everything that survived. The work is repetitive, the criteria are written down in advance, and the decisions are supposed to be reproducible. That combination is exactly what a language model can assist with, and it is why the screening step has attracted so much tooling. MetaScreener is aimed at review teams who already have criteria and a search export, and who want a first pass that produces include/exclude calls with a confidence number attached, so that only the uncertain fraction reaches a human. The README frames the pitch as replacing a single model with an ensemble: multiple LLMs run in parallel and their outputs are aggregated through what the project calls a calibrated confidence pipeline. The stated output is not just a label but a decision plus a confidence score plus a tier, with uncertain cases routed to human review. That routing is the part that matters. A tool that auto-decides everything is unusable in a review context; a tool that flags its own uncertainty is at least auditable.

The four-layer Hierarchical Consensus Network

The architecture diagram in the README describes four layers. Layer 1 is inference: four or more LLMs are called in parallel through API providers, and the README names DeepSeek, Qwen, Llama and Kimi among the supported families. Layer 2 is a rule engine that sits between the models and the aggregation. It applies hard rules that auto-exclude a record and soft rules that apply score penalties. This is the layer most likely to be under-documented in practice, because the behaviour of a hard exclude depends entirely on how the rules were written for your protocol. Layer 3 combines two mechanisms: Calibrated Confidence Aggregation (CCA) and Element Consensus Scoring (ECS). ECS scores agreement per PICO element rather than per paper, so a record can show strong agreement on population and weak agreement on outcome. Layer 4 is a router with four tiers: Tier 0 for hard-rule exclusion, Tier 1 for high-confidence auto-decisions, Tier 2 for moderate auto-decisions, and Tier 3 for human review. The data flow is one-directional: records enter from an .ris, .bib, .csv or .xlsx upload, pass through the layers, and land in a decision store with a tier label. The README also states that inference uses temperature=0.0 and seed=42, which is the standard attempt at determinism for hosted models. It is worth being clear about the limit of that: a seed parameter constrains sampling, not the upstream model weights, so determinism holds only as long as the provider serves the same model version.

Calibration and the active learning loop

The confidence numbers are the most interesting design decision here. Raw LLM scores are not probabilities, and the README is explicit that MetaScreener applies post-hoc calibration using Platt scaling or isotonic regression to map raw scores onto estimated true probabilities. That is a meaningful step: it means the Tier 1 and Tier 2 thresholds are set against a calibrated scale rather than against an arbitrary score cut. The second mechanism is an active learning loop. Human feedback on screened records is fed back to recalibrate model weights in real time, according to the README. In practice this means the tool expects you to review some records and correct them, and it uses those corrections. The trade-off is that calibration is only as good as the labelled sample behind it. A review team that runs screening straight through without touching the Tier 3 queue gives the calibration nothing to work with. There is also a sequencing problem: the calibration is fitted on decisions the ensemble already made, so early rounds of screening are operating on the least-calibrated version of the pipeline. Teams that treat the first hundred records as a calibration set and the rest as production will get more out of this than teams that expect the default thresholds to be correct on day one.

Getting it running: pip, Docker, or source

There are three documented install paths. The simplest is pip: run `pip install metascreener`, then `python -m metascreener`, and the README says the web UI opens at http://localhost:8000. The Docker path avoids Python entirely: `docker pull chaokunhong/metascreener:latest`, then `docker run -p 8000:8000 -e OPENROUTER_API_KEY="sk-or-v1-your-key-here" chaokunhong/metascreener`, again on port 8000. The source path is for developers and needs uv and Node.js 18 or later: `git clone https://github.com/ChaokunHong/MetaScreener.git`, `cd MetaScreener`, `uv sync --extra dev`, then `python run.py`, which the README says starts the FastAPI backend on port 8000 and the Vite frontend on port 5173. All three paths require an OpenRouter API key, which you can set either in the Settings page of the web UI or via the `OPENROUTER_API_KEY` environment variable. Python 3.11 or higher is required. The workflow is a numbered sequence in the UI: criteria, settings, title/abstract screening, full-text screening, extraction, quality assessment, evaluation, history. Screening inputs are .ris, .bib, .csv or .xlsx files. The README's quick-start ends mid-sentence in the screening flow section, so the exact review screen behaviour past the per-paper decision, confidence, tier and per-element consensus display is not documented in the material available.

Where the ensemble approach breaks down

The cost claim is the first thing to interrogate. The README gives roughly $0.003 to $0.009 per paper depending on preset, with Balanced at about $0.005 and Precision at about $0.009. Those figures are per paper and per pass, and a review has two passes. A 5,000-record title/abstract screen at the Balanced rate is around $25, which is cheap. The full-text pass is where the arithmetic changes, because full-text screening involves chunking PDFs and the token volume per record is far higher than an abstract. The README does not give a per-paper cost for the full-text stage, so the headline number should not be extrapolated to it. The second limitation is model drift. The supported-model table lists fifteen models across three tiers, all served through OpenRouter. You do not control those endpoints. A provider can deprecate a model, change a version, or alter serving behaviour, and any of those events invalidates a temperature=0.0, seed=42 run. The project does ship an audit trail and a History page with decision provenance, which is the right mitigation, but it only helps if the provenance record includes the resolved model identifier rather than the display name. The third limitation is domain fit. The README claims strong medical knowledge for the flagship tier, but calibration is fitted on data, and a review in an unusual domain will need its own labelled sample before the Tier 1 threshold means anything. The fourth is that the tool screens, it does not adjudicate. Tier 3 records still need a human, and the proportion that lands in Tier 3 is not stated in the material.

How this differs from single-model and rule-based screening

The obvious comparison is a single general-purpose LLM called directly against your criteria, which is what most teams build first. The difference is not accuracy in the abstract; it is failure visibility. A single model that is confidently wrong gives you no signal. An ensemble with per-element consensus gives you a disagreement score, and disagreement is the thing you can act on. The second comparison is to classical screening automation, the term-frequency and classifier-based approaches used in tools that rank records by relevance. Those methods are cheap, fast, and entirely local, and they work well for prioritising a large set. What they cannot do is read a PICO criterion written in prose and apply it. MetaScreener's rule engine plus LLM inference is doing semantic matching against written criteria, which is a different task from ranking. The third comparison is to commercial screening platforms. The relevant difference is the licence and the deployment model: MetaScreener is Apache-2.0 and can be run from source or from a container you host, with the model calls going out to a provider of your choosing. That matters for teams with data-handling constraints, though it does not remove the constraint, because the records still leave your infrastructure for the inference call. If your records cannot leave your network at all, none of the documented install paths work as described.

Licence, maintenance, and what the release history shows

MetaScreener is licensed Apache-2.0, which permits commercial use, modification and redistribution provided the licence and notices are preserved. That is a permissive choice and it removes the licensing question for most institutional deployments. It does not address the data question, which is separate: sending records to OpenRouter is a data-transfer decision governed by your ethics approval and your provider's terms, not by the software licence. On maintenance, the release history is active but pre-stable. The most recent release shown is fp-audit-protocol-v1.0 from May 2026, which the release name describes as a false-positive audit protocol lock. Before that come v2.0.0a4 and v2.0.0a3 from February 2026, both alpha versions. The pattern suggests the v2 line is still in alpha while the project pins down its evaluation methodology. For a review team, that means pinning a specific version rather than tracking latest is the sensible default, and it means the audit protocol is the part of the project most likely to change under you. The practical upgrade cost is low if you stay on the pip or Docker path, since both are single-command updates, and higher if you run from source, because that path pulls in a Node.js frontend build alongside the Python backend. There is no migration documentation in the material for moving a screening session between versions, so treat the History page as the record of what a given version decided.

Editorial conclusion

MetaScreener is worth adopting if you run high-volume title/abstract screening and already have a written PICO/PEO/SPIDER protocol and a human reviewer budget for Tier 3 cases. It is the wrong tool if your review must run fully offline, if you cannot send records to OpenRouter, or if you need a screening decision that is defensible without any human adjudication. Before committing, verify three things: that the Balanced preset's four models are actually reachable with your key, that the calibration step produces sensible probabilities on a held-out sample from your own domain (not just the built-in evaluation page), and that the audit trail in the History page records model version strings, because a silently swapped upstream model invalidates the reproducibility argument.

Official sources

1. ChaokunHong/MetaScreener on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases