Arctic Shift: Reddit dumps, API and web search

The archive is the product, the repository is the index

Reddit's own data access is not designed for bulk historical work, and the alternatives that filled that gap have been unstable. Arctic Shift names one of its own repository topics directly: pushshift, the long-standing third-party Reddit archive. Its stated goal is making Reddit data accessible to researchers, moderators and everyone else.

What it actually is, judging from the repository layout, is a collection plus tooling rather than an application. The top level contains `download_links.md`, `file_content_explanations.md`, an `api/` directory, a `schemas/` directory and a `scripts/` directory. That is an index of dumps, a description of what is inside them, some code, and schema definitions, and almost nothing else.

That shape has a practical consequence. There is no package to install and no library to import. If you want the data, you download a dump, or you query someone else's server, and you write the parsing yourself. The value is in the bytes, not in the code.

The audience in the project description is worth repeating because it is broader than it looks: researchers, moderators, and everyone else. Moderators looking at what was said in their own subreddit over time are a different use case from an academic, and the project ships a per-user and per-subreddit download tool for the smaller cases.

Monthly dumps, and what happens between months

Dumps arrive as monthly releases. Recent tags are 2026_09 dated 2026-10-11, 2026_08 dated 2026-09-12 and 2026_07 dated 2026-08-09. The gap between the month named in the tag and the date of the release is roughly a week, which tells you something about the cadence: this is a monthly archive built after the fact, not a live feed.

Download links live in `download_links.md` in the repository, and the README notes that each new dump is also added to the releases page. Two locations to check, and they do not necessarily carry the same things.

The file formats matter more than the release names. The helper scripts can process `.zst`, `.zst_blocks`, `.jsonl`/ndjson and `.json`. That is a compressed archive rather than a database export, which is the right call at Reddit's volume and the wrong call if you expected to point a SQL client at it.

The README's advice is direct: work with the compressed files instead of unpacking them, unless you have what it calls seemingly infinite disk space. For most analyses you will decompress a stream inside your own code rather than materialising files, and the Python scripts in `scripts/` are written on that assumption.

Three ways in: dumps, API, web search

The project gives you three doors and they suit different situations.

The bulk dumps are the ground truth. The API is described by its author as limited, and the README does not document rate limits, quotas, or a stability guarantee for it. The manual query path is a web tool at arctic-shift.photon-reddit.com/search, which the README pairs with a download tool for pulling data on specific users or smaller subreddits.

The honest ranking for a technical user is: dumps for anything reproducible or large, API for quick exploration, web search for one-off lookups. Inverting that order, building a pipeline on the API and assuming it will keep serving, is the risk to avoid, because the archive is maintained by one person and describes that endpoint as limited in the same sentence that offers it.

There is one more interface that matters for compliance. The README links removal request forms for Arctic Shift, Pushshift, PullPush and archive.org, and directs you to search the dataset for your own username first. The project takes takedown requests seriously enough to publish a form and to point at the equivalents for the archives it resembles. That is a good sign about the maintainer's intent, though it is not the same thing as a contractual deletion guarantee.

The Python helper scripts are a starting point, not a library

The README's usage section is a six-step tutorial, and it is worth reading as a description of the intended workflow rather than a set of commands to memorise.

You clone the repository with its submodules, which is why `--recursive` is there:

git clone --recursive https://github.com/ArthurHeitmann/arctic_shift.git
cd arctic_shift

Then you install the one decompression library the scripts rely on, which the README calls `zstandard`:

pip install zstandard

Python 3.10 is the stated minimum.

What follows is the part that tells you the most about how this project expects to be used. You open `scripts/processFiles.py` in an editor. You set `fileOrFolderPath` to a file or a directory, and because it is a raw string you do not have to escape backslashes for Windows paths. You write your own code into the `processFile` function. Then you run it and, in the author's phrasing, be very patient.

That is a template, not an SDK. You are expected to paste your own logic into someone else's function and run it over potentially enormous files. It is a reasonable way to get a first result in an afternoon and a poor foundation for a pipeline you intend to maintain.

The gaps you have to fill yourself

Several things a researcher would ask are simply not answered, and knowing that before you start saves weeks.

There is no licence. The repository metadata reports no licence identifier, and the README has no licence section either. That is the most consequential omission here. You may read the public dumps, but you have no stated permission to redistribute them, publish derived datasets, or ship them inside a product. For an archive built by running large-scale collection against another company's platform, that question is not academic. Ask, in writing, before you build anything on top of it.

There is also no schema guidance in the README itself, only a pointer to `file_content_explanations.md` for how the data was collected and modified, and a `schemas/` directory for structure. That distinction matters: understanding what a field means, and how it was produced, is a different task from knowing its type.

Then there is the durability question. The archive is maintained by one person, reachable by Discord, email, GitHub issues or donations through PayPal and Buy Me a Coffee. That is not a criticism, it is a risk statement. If the project stops, there is no organisation behind the dumps you already downloaded, and no guarantee the hosted API and web tool outlive the repository.

When to use it, and when to look elsewhere

The case for Arctic Shift is scale and history. If you need Reddit data at scale, the monthly dumps may be useful.

The case against is when you need freshness or guarantees. Data from the current month may not yet be available, depending on release timing. There is no uptime story for the API and no stated limit on what it will serve. And for anything involving personal data at scale, the removal-request process is evidence that the archive contains things people have asked to be removed, which is a fact you need to account for in your own retention policy rather than a detail to skip.

One last practical note on the workflow. Processing is described as slow, and it is streaming over compressed archives rather than querying a database. Budget for that on the first run, not after you have committed to a deadline, and start with the download tool on a single user or small subreddit rather than cloning a full month before you know whether the schema suits your question.

Editorial conclusion

Arctic Shift suits a researcher or moderator who needs Reddit data at scale, and who is prepared to handle zstandard-compressed files and wait for processing to finish. It does not suit a product team that needs a licence, a service level agreement, or a guarantee that last month's dump lands on the same day, because the repository declares no licence and releases are produced by one person who describes the API as limited. Before writing a line of analysis code, ask the maintainer what licence applies to the dumps and the API, check that the months you need exist in the releases, and pin your own copy of the data rather than depending on the hosted endpoints.

Frequently asked questions

Official sources

1. ArthurHeitmann/arctic_shift on GitHub
2. Issues
3. Project website
4. README
5. Releases