paperscraper, one interface for publication metadata across five servers

Five sources, one interface, and a plotting library on top

paperscraper is a Python package for scraping publication metadata or full text files, PDF or XML, from PubMed or from preprint servers such as arXiv, medRxiv, bioRxiv and chemRxiv.

The list of capabilities in the README is what makes it more than a set of scrapers. It provides a single interface to scrape metadata, retrieve citation counts from Google Scholar or Semantic Scholar, query journal impact factors, and run simple postprocessing and plotting routines for meta-analysis.

So the shape of the tool is a data collection layer plus a small analysis layer. That combination is unusual in this ecosystem, where the norm is either a search wrapper or an analysis package, and it reflects the use case of a researcher doing a systematic review: gather the literature, then count citations, then attach journal metrics, then draw the figures.

The audience is academics and researchers. The package classifiers say it plainly, with the intended audience listed as both Developers and Science/Research, and the development status is 5, Production/Stable.

The dependency list explains the architecture better than the prose does, because paperscraper is not built from scratch. The core work is delegated to `arxiv`, `pymed-paperscraper` and `scholarly`, and the analysis side is pandas, seaborn, matplotlib and matplotlib-venn for the Venn diagrams a meta-analysis needs, with bibtexparser for bibliography handling and thefuzz for fuzzy matching. The impact-factor package handles the journal metrics, semanticscholar is the second citation source, and thefuzz and tldextract do the fuzzy matching and domain extraction that reconciling records across servers requires.

That last point is the real difficulty in this problem, and it is why those two small libraries are there. The same paper appears in PubMed with one title, on arXiv with another, and in a preprint server with a third, and merging them into one record set is where most of the actual work sits.

The licence is MIT, the homepage is a documentation site at jannisborn.github.io/paperscraper, and the authors are Jannis Born and Matteo Manica.

Installation, and the local development setup

Installing is a single command, and the project also supports uv.

pip install paperscraper

uv add paperscraper

That is enough to query PubMed, arXiv or Google Scholar, which is the claim the README makes immediately after the install instructions.

For local development the project uses uv throughout, and the sync command installs the project and the development tooling into a `.venv`.

uv sync

uv run python -c "import paperscraper"

The second command is the quick check that the import works in the environment uv created, which is a habit worth having for any package with this many dependencies.

The packaging configuration is modern and the constraints are worth knowing. The build backend is setuptools with `setuptools>=68` and wheel required, the version is dynamic, and the supported Python range is `>=3.9,<3.14`, so 3.9 through 3.13. The classifiers list each of those versions, which is the usual way of saying the package is tested on all of them.

The dependency list is long but each entry has a clear job, and a few are worth calling out because they shape the behaviour you will see.

`arxiv>=2.4.0` and `pymed-paperscraper>=1.0.6` are the two scrapers for the primary sources, so the fork of pymed is maintained specifically for this project. `scholarly>=1.0.0` and `semanticscholar>=0.8.4` are the two citation sources. `impact-factor>=1.1.3` is the journal metric lookup. `kaggle>=1.7.4.5` is the arXiv bulk download backend, `boto3` and `dotenv` are there for credential and configuration handling, `pydantic` for data validation, `unidecode` for transliterating author names, and `bs4` for HTML parsing.

The `pytest` entry sits in the runtime dependency list rather than in a development extra, which is a small packaging inconsistency but also a hint that the test suite is run against the installed package.

The repository is a MkDocs site with a `mkdocs.yml` at the root, a `docs/` directory, and a `paper.md` and `paper.bib` pair, which is the convention for a paper describing a software tool. There is also a `GOVERNANCE.md` and a `SUPPORT.md`, a `.coveragerc` and a `codecov.yml`, a `.pre-commit-config.yaml` and a `.codespellrc`.

The preprint servers are searched through local dumps

The most important design decision in the package is that bioRxiv, medRxiv and chemRxiv are not queried live. The entire history of papers is downloaded and stored in the `server_dumps` folder in JSONL format, one paper per line, and your searches run against that local file.

The three dump functions are called with no arguments for a full history.

from paperscraper.get_dumps import biorxiv, medrxiv, chemrxiv
chemrxiv()  #  Takes <15min -> +50K papers (~30 MB file)
medrxiv()  #  Takes <5min -> +100K papers (~200 MB file)
biorxiv()  # Takes <1h -> +450K papers (~800 MB file)

The comments give the cost directly. chemRxiv is the smallest at around 50 thousand papers and 30 megabytes, taking under fifteen minutes. medRxiv is around 100 thousand papers and 200 megabytes in under five minutes, which is faster than chemRxiv because it is smaller to transfer. bioRxiv is the big one, around 450 thousand papers and 800 megabytes, taking up to an hour.

So the trade is a one-off download of a few hundred megabytes in exchange for local search over the whole corpus, with no per-query network cost and no rate limit from the server. For a researcher running hundreds of queries across a systematic review, that is clearly the right way round.

The date-filtered variant is where the sharp edge is, and the README warns about it twice.

medrxiv(start_date="2023-04-01", end_date="2023-04-08")

The warning is that the resulting `.jsonl` file is labelled according to the current date, and all your subsequent searches will be based on this file only. So a date-filtered download does not add to the corpus, it replaces it, and a later query that expects a paper from 2019 will not find it. The advice is to keep an eye on the source files under `paperscraper/server_dumps/*jsonl` to make sure they hold the metadata for everything you care about, and there is a helper for inspecting the active dump directory, `paperscraper.utils.get_server_dumps_dir()`.

There is also a note to restart the Python interpreter after the dumps are stored, so the changes take effect, which is a symptom of the state being held in module-level state that is read at import time.

The retry behaviour is configurable, which matters for the big downloads. The parameters are `max_retries`, a `request_timeout` given as a connect and read timeout pair, a `retry_backoff_seconds` for the initial backoff, `max_workers` for the number of parallel date windows, and `window_days` for the size of each window. The comment notes that smaller windows increase parallelism, so the tuning knob is a trade between request size and request count.

arXiv has two bulk backends, and the default is the blunt one

Local search can be faster than the arXiv API, especially if you plan many queries, and paperscraper therefore provides two backends to bulk-download arXiv, Kaggle and the `arxiv` package, with the default being Kaggle because it is much faster.

The Kaggle backend needs an authenticated account, and the setup is a single command.

kaggle auth login

The download function takes a date range.

from paperscraper.get_dumps import arxiv
arxiv(start_date='2019-01-01', end_date='2026-12-31')

And here is the cost, stated in the README: the disadvantage of the Kaggle backend is that it bulk-downloads all of arXiv. The note is direct that for small API-backed dumps you should use the `arxiv` PyPI package backend instead, and gives the call.

from paperscraper.get_dumps import arxiv
arxiv(start_date='2024-01-01',end_date='2024-01-04',backend='api')

The difference in approach between the two backends is instructive. Kaggle distributes the full corpus as a dataset, so you get everything at once and the transfer is optimised for bulk data. The API backend makes the ordinary per-query requests against the arXiv API for a date range, which is slower in aggregate but gives you only the window you asked for.

Afterwards you search the local arXiv dump the same way as the other preprint servers, so the corpus is uniform across sources. The direct endpoint is named in the README, `paperscraper.arxiv.get_arxiv_papers_local`, and the backend can also be specified directly in the `get_and_dump_arxiv_papers` function.

from paperscraper.arxiv import get_and_dump_arxiv_papers
get_and_dump_arxiv_papers(..., backend='local')

The `backend='local'` value here selects the local search path rather than a dump backend, which is a slight inconsistency in the naming that is worth knowing if you read the code.

Google Scholar, and why the sponsorship badge matters

The most consequential thing in the README for anyone planning a long-term project is the Google Scholar story, and it is not entirely visible from the feature list alone.

The README carries a badge saying the project is supported by SearchApi, with a link to their Google Scholar API and the parameters of that sponsorship, and an image inviting the reader to get an API key. The announcement says the project is happy to announce that paperscraper now supports scraping Google Scholar via SearchApi, and points to a Scholar Metrics Analysis guide in the documentation for examples.

That is a meaningful change in the dependency surface. The `scholarly` package in the dependency list is the well-known open source scraper for Google Scholar, and it works by making requests as a browser would. Google Scholar has no official public API, and using an unofficial scraper at scale is exactly the sort of thing that gets an IP blocked, which is why a paid third-party API with an official contract is a sensible route for a project that wants to keep working.

The practical consequences for a user are three. There is now an API key to obtain and configure, and a cost that depends on the provider's pricing. The access is mediated by a third party, so your queries and the terms under which it is provided are the provider's. And the code path is different from the scholarly one, so the behaviour of the citation counts can differ between the two sources.

That last point is not a criticism but a caution for anyone doing analysis on citation counts. The README lists both Google Scholar and Semantic Scholar as sources, and the two do not agree, because they count citations differently, cover different corpora and update at different times. The tool offers both, which is the right design, and choosing one and stating it is the reader's responsibility.

The version history reflects the shift. The recorded releases are v1.1.0 from 2026-10-08, titled with Google Scholar integration, v1.0.0 from 2026-06-16, titled as the stable release, and v0.3.6 from 2026-03-17. So the 1.0 milestone was a few months ago and the Scholar integration landed in the release after it.

The last push to the repository was on 2026-10-14, and the example guides in the documentation cover paper keyword analysis, PDF retrieval, Scholar metrics analysis and self-citation analysis, which are the four workflows the project is organised around.

Where the alternatives sit

The tools this package builds on are all in the dependency list, and each covers part of the problem, so the honest comparison is with using them directly.

The `arxiv` package does arXiv. The `pymed` fork does PubMed. The `scholarly` package does Google Scholar, and `semanticscholar` does Semantic Scholar. What paperscraper adds is the layer above: a single interface across all of them, the local dump mechanism for the preprint servers, the record reconciliation across servers, the impact factor lookup, and the plotting layer with Venn diagrams for the overlap between result sets.

That reconciliation is the part that cannot be skipped in a meta-analysis, and it is where the effort has gone. When you run a search on PubMed and the same search on arXiv, you get two lists with different titles, different author name formats and different identifiers, and the number of unique papers is not the sum and not the maximum. A Venn diagram of the overlap is the standard way to present that, and matplotlib-venn being a dependency is a sign that the package expects you to produce that figure rather than only count it.

The alternative approaches are three. Using the underlying libraries directly is the obvious one, and it means reimplementing the dump mechanism, the reconciliation and the figures yourself. Using a commercial bibliographic database API is the second, which gives you structured data with a contract and a cost, and which is what a pharmaceutical company or a large medical institution would actually do for a systematic review, because the provenance and the coverage guarantees matter when the output is a regulatory document. And the third is the traditional route, a librarian-run search with a proper database, which is what a clinical guideline still requires.

The value paperscraper adds is being free, open source, scriptable and able to run offline against a snapshot once the dumps are downloaded. That makes it a good fit for teaching, for exploratory analysis, for a lab with no budget for a database subscription, and for anyone who wants to run the same query against a fixed corpus and get the same answer every time. It is a poor fit for a regulatory systematic review, where the search has to be documented, reproducible and defensible, and where the completeness of the source is a claim you need a provider to stand behind.

Editorial conclusion

paperscraper suits a researcher or analyst who needs one API over several literature sources, who runs many queries and therefore wants local search rather than per-query requests, and who needs meta-analysis figures without writing the plotting code. It does not suit a project that needs a strictly live view of a preprint server, because the xRxiv path searches a snapshot. Verify first which Scholar backend you are using and what it costs, since Google Scholar access now goes through a sponsored third-party API, that the dump you search is the one you think it is, since a date-filtered download produces a file that replaces the full history for later searches, and that the Python version you run is inside the supported range of 3.9 to below 3.14.

Frequently asked questions

Official sources

1. jannisborn/paperscraper on GitHub
2. License: MIT
3. Project website
4. README
5. Releases