Auto Archiver: turning a list of URLs into a verifiable archive

One URL in, one archived and enriched item out

The problem Auto Archiver solves is bulk. There is a link in a spreadsheet, a cell in a shared document, a row in a CSV, and somebody has to fetch it, keep what it pointed to, and prove later what was collected. Doing that by hand for a handful of links is fine. Doing it for hundreds, across several platforms, with a written record of each result, is the kind of work that quietly eats a week.

The tool takes URLs from different sources, a CSV file, Google Sheets, the command line, and archives the content of each one. The content types it names are social media posts, videos, images and webpages. What comes out the other side is not just a file: the content is enriched, then saved either locally or remotely, to an S3 bucket or Google Drive.

Two design choices stand out for an engineer reading the source. The orchestration is configuration driven, through an `orchestration.yaml` file that names the modules to use, and the tool is a service rather than a script, so the same configuration can be run repeatedly. And the reporting is bidirectional: the status of the archiving process can be appended to a CSV report, or, when the input came from Google Sheets, written back to the original sheet. The sheet is the work queue and the sheet is the report, which is the shape an investigation actually wants.

Platform support is documented separately, because it changes constantly

The repository carries a `PLATFORM_STATUS.md` file and a machine-readable `platform_status.json` at the top level. That pairing tells you something about how the project is maintained: support for individual sites changes as those sites change, so it is tracked apart from the main documentation and kept in a form a program can read.

This is the honest limit on what the tool can do. An archiver that scrapes social platforms depends on those platforms not changing their markup, their endpoints or their access rules, and any of them can. A separate status file lets a user check before they commit a batch, which is a much better failure mode than discovering it halfway through a hundred rows.

The dependency list explains why the project needs its own crawler rather than only making HTTP requests. Alongside the usual request and parsing libraries, there is `yt-dlp` for video, `instaloader` and `telethon` and `python-twitter-v2` for social platforms, `seleniumbase` and `pyautogui` for pages that need a real browser, and `warcio` for writing WARC output. The `ffmpeg-python` binding and an `exiftool` installation in the container point at media handling, and `pdqhash` and `pillow` at image work. The project is a stack of specialised fetchers behind one orchestrator, and each one carries its own fragility.

Verifiability is a dependency choice, not a promise

The project describes itself as archiving in a secure and verifiable way, and the dependency list is where that claim gets concrete. `rfc3161-client` and `opentimestamps` are there for trusted timestamping, `cryptography` for the cryptographic operations, and `hashlib`-style image fingerprinting through `pdqhash` for content matching rather than filename matching.

That combination points to a specific idea of evidence handling. A timestamp proves when a piece of content was captured, not just that it exists now, and a perceptual hash lets you detect the same image reposts with a different name. Neither of those is possible with a plain download, and neither is free: timestamping services need network access, and a timestamp that cannot be validated later is worth much less than one that can.

The output formats back this up. `warcio` writes WARC files, the archival format used for web captures, and the local storage path writes into a `local_archive` directory. The tool is aiming at a workflow where a capture is later verifiable, which is the difference between a copy of a file and a record of evidence.

Running it from Docker or from pip

The quick start in the documentation is a single Docker command that pulls the published image and points it at a configuration file:

docker pull bellingcat/auto-archiver && docker run -it --rm -v secrets:/app/secrets bellingcat/auto-archiver --config secrets/orchestration.yaml

The `secrets` volume mount is the important part. The configuration file names the modules to run, and some of them need credentials, for instance to reach a Google Sheet or a private S3 bucket, so the config directory is mounted rather than baked into the image. The `--config` argument then tells the container which orchestration file to read.

If you would rather install it into a Python environment, the pip route is one line:

pip install auto-archiver && auto-archiver --help

Both routes run the same CLI, so the `--help` output is the fastest way to see the options the installed version actually supports.

There are constraints worth knowing before you pick a route. `pyproject.toml` requires Python 3.10 or newer and below 3.13, so a system Python outside that window will not install cleanly. The project is built with poetry, uses a `poetry.lock` file, and the Docker image handles that for you by creating a virtual environment inside the image.

The compose file shows what the container expects on disk

The `docker-compose.yaml` in the repository is small and tells you how the container is meant to be wired into a working setup:

services:
  auto-archiver:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: auto-archiver
    user: "${USER_ID:-1000}:${GROUP_ID:-1000}"
    volumes:
      - ./secrets:/app/secrets
      - ./local_archive:/app/local_archive
    command: --config secrets/orchestration.yaml

Two volumes, and the reason for each is clear. `secrets` holds the orchestration file and the credentials the modules need. `local_archive` is where captures land, and it is a host directory so that the archive outlives the container.

The `user` line is the part worth copying. It defaults to 1000:1000, and the comment explains why: matching the host UID and GID avoids permission problems on the mounted volumes. The Dockerfile also chowns the archive, secrets and temp directories to that user, and the image runs as non-root. The `USER_ID` and `GROUP_ID` environment variables are the override, and the defaults are 1000 for both.

For a long running service, `docker compose` from the repository root builds the local Dockerfile rather than pulling the published image, which is what you want if you are testing a change. The Makefile wraps the common cases as `make docker-build`, `make docker-compose` and `make docker-compose-rebuild`.

The container inherits a browser crawler and installs ffmpeg and exiftool

The Dockerfile does not start from a plain Python base. Its first line is `FROM webrecorder/browsertrix-crawler:1.13.2 AS base`, so the image it produces is a Browsertrix Crawler image with the Auto Archiver Python application layered on top.

That choice is the reason the image is as heavy as it is. A browser-based crawler brings a browser, a driver and a display stack, and the Auto Archiver layer then installs its system dependencies with apt-get: gcc, ffmpeg, fonts-noto, exiftool and python3-tk.

The fonts-noto package is not cosmetic. Media archiving and OCR of rendered pages both break in surprising ways when the container has no fonts, and exiftool is what pulls metadata out of images and video containers. ffmpeg is what lets the tool do anything with the media it fetches.

The Python side is then installed with poetry into a virtual environment at `/app/.venv`, with the dependencies installed from `poetry.lock` before the source is copied, so that a code change does not invalidate the dependency layer. The entry point runs the application directly from the virtual environment on the PATH.

Where Auto Archiver is the wrong tool

There are two clear cases where it is the wrong choice.

The first is a single file. If you need one video, `yt-dlp` on its own is a fraction of the setup, needs no orchestration file, no secrets volume and no container, and does not need a Google service account to write a report back to a sheet. Auto Archiver pays for itself in orchestration, enrichment and reporting, and that overhead is only worth paying at a scale where those things matter.

The second is the display dependency. The dependency list includes `pyautogui` and the container installs `python3-tk`, both of which involve graphical interfaces, while the base image is a headless browser crawler. The Dockerfile does not document a display or a virtual framebuffer for those paths, so if you enable an enricher that drives a graphical browser, you are likely to hit environment setup that the standard compose file does not cover.

There is a third, quieter cost. The project pins `numpy` to an exact version, `numpy (==2.1.3)`, in a tool with a long dependency list that includes `yt-dlp`, `seleniumbase` and `cryptography`. Exact pins are the right call for reproducibility, and they are also the thing that will need attention when a dependency needs to move.

Compared with a crawler alone, or a scraper alone

The honest alternatives are not competing archiving products but the two pieces Auto Archiver is built from.

Browsertrix Crawler, the image the container starts from, is a crawler you could also use on its own. That is a smaller tool with a smaller dependency tree and a simpler deployment, and it is the better choice when your inputs are ordinary web pages and the output you need is the capture. What you give up is the enrichment stage, the choice of storage backend, the enrichment of metadata beyond the capture itself, and the write-back of per-URL status into a sheet or a CSV report. Auto Archiver is that tool wrapped in an orchestrator, a storage abstraction and a reporting loop.

The other alternative is a hand-written scraper using `yt-dlp`, `instaloader` and a requests session, which gives exact control over each platform and no browser at all. Auto Archiver wins when the links come from many platforms, when the result has to be stored in a place with a retention policy such as S3, and when someone else has to be able to re-run the batch later from a config file. The scraper wins when one platform matters most and you need behaviour the tool does not offer.

The version history shows the direction of travel. Release 1.2.6 is annotated as a ghost archive enricher plus dependency updates, and 1.2.9 is the current tag, released on 2026-09-01. The last push to the repository was on 2026-09-10, and the project is MIT licensed, with the licence declared in `pyproject.toml` and a `LICENSE` file at the root.

Editorial conclusion

Auto Archiver is a good fit for investigations and archival projects where a column of links has to become a folder of evidence with a record of what succeeded, and a poor fit for a one-off download where yt-dlp on its own is quicker. Before adopting, read PLATFORM_STATUS.md to see which sites are actually supported, and pin the Python version, because pyproject.toml restricts the project to 3.10 up to, but not including, 3.13 and pins numpy to an exact build.

Frequently asked questions

Official sources

1. bellingcat/auto-archiver on GitHub
2. License: MIT
3. Project website
4. README
5. Releases