brozzler: a distributed crawler that drives a real browser

A crawler that renders the page before it reads it

brozzler is a distributed web crawler from the Internet Archive that uses a real browser, Chrome or Chromium, to fetch pages and embedded URLs and to extract links. The name in the project documentation is a pun on the two halves of that design, browser and crawler.

The difference from an ordinary crawler is the whole point. A client that issues plain HTTP requests sees the server's response and nothing else, so it never sees a link that a script inserted into the page after load. brozzler hands the page to a browser, lets the page run, and takes the links from the result.

Two supporting pieces come with it. yt-dlp, the successor to youtube-dl, is employed to enhance media capture capabilities, so a page with an embedded video is handled by a tool that understands media extraction rather than by the browser alone. And warcprox, the Internet Archive's own recording proxy, is what the crawler is designed to work with, since the point of the whole pipeline is to produce an archive rather than a search index.

The requirements are modest on paper: Python 3.9 or later, a RethinkDB deployment, and Chromium or Google Chrome at version 64 or newer. The manifest in the repository is stricter on Python, requiring 3.10 or newer, which is a small discrepancy worth noticing if you are choosing an interpreter.

The audience is archiving institutions and research groups rather than SEO crawlers, and the deployment shape is a service, not a script.

RethinkDB for state, warcprox for the archive

The two server-side dependencies define the architecture, and it helps to keep their roles apart.

RethinkDB manages the crawl state. It is the queue, the record of what has been captured and what is still to be fetched, and it is what makes the crawler distributed: workers share state through the database rather than through a coordinator, so adding capacity means adding workers.

warcprox is the recording proxy. Brozzler is designed to work in conjunction with it for web archiving, and the crawler is not a standalone archiver on its own. The README documents a convenience option for this pairing, the --warcprox-auto flag, which tells brozzler to look for a healthy warcprox instance in the RethinkDB service registry. For that to work you need at least one warcprox running with the --rethinkdb-services-url option pointing at the same RethinkDB services table that brozzler is using, and the documentation recommends the flag for clustered deployments.

That last point is the operational detail that catches people. If your workers and your warcprox instances look at different services tables, the workers will crawl and record nothing, and the symptom is an empty archive rather than an error.

A third piece, doublethink, appears in the optional extras alongside rethinkdb itself, which is what makes the service registry lookup work.

From one URL to a worker fleet

The documented entry point is deliberately small. The simplest way to get started is the brozzle-page command line utility, which passes in a single URL to crawl:

pip install brozzler[rethinkdb]

brozzle-page https://example.org

For anything more, you define a job in a YAML file and start a worker. The install documentation lists the whole local workflow on macOS, starting the database with brew install rethinkdb and then running rethinkdb in the background, optionally creating a virtualenv, installing the package with the rethinkdb extra, and then either crawling a single site or enqueueing a job and starting a worker:

brozzler-new-job job1.yml
brozzler-worker

The production entry points are the same commands with different arguments:

brozzler-worker --warcprox-auto
brozzler-new-job myjob.yaml
brozzler-new-site --time-limit=600 https://example.org/

That last command is worth noting. brozzler-new-site submits a site that is not tied to a job, with a time limit of 600 seconds, which is the quickest way to bound the cost of a crawl of something that might turn out to be very large.

The documentation also makes a useful admission about this mode. Running brozzler locally in this way demonstrates the full archival crawling workflow but does not take advantage of the distributed nature. The distributed part is workers sharing RethinkDB state, which only becomes interesting once several of them run against the same database.

The command line entry points are listed as console scripts in the package manifest, alongside brozzler-ensure-tables for creating the database schema and brozzler-list-captures for listing what has been captured.

Job files carry seeds, limits and scope

Jobs are YAML, and the configuration is small enough to read in one screen.

    id: myjob
    time_limit: 60 # seconds
    ignore_robots: false
    warcprox_meta: null
    metadata: {}
    seeds:
      - url: https://one.example.org/

Options may be specified either at the top level or on individual seeds, and at least one seed URL is required while everything else is optional. Each seed can override the top level, so a single seed can carry its own time limit, as the example shows with a 30 second limit on the second seed, its own ignore_robots setting, and its own scope.

Three of those options carry the policy. time_limit bounds how long a seed is crawled, which is the main brake on an expensive crawl. ignore_robots defaults to false, meaning a job respects robots.txt unless you explicitly say otherwise. And scope uses a surt prefix, in the example https://(org,example,, to keep the crawl inside a domain.

The surt form is the SURT prefix format, a canonical, sortable representation of a URL used in web archiving, and it is a better way to express a domain boundary than a regular expression. A job also carries warcprox_meta and a free-form metadata mapping, which is how per-crawl metadata such as collection identifiers reaches the archive record.

Two more configuration points exist outside the job file. The workers need to know where warcprox lives, and the crawler needs to know which database to use, both of which the documentation covers in the sections referenced from job-conf.rst and the command line help.

A small Flask dashboard for crawl status

Brozzler ships with what the documentation calls a rudimentary web application for viewing crawl job status.

It is an optional extra rather than part of the core, so the install is a separate step:

pip install brozzler[dashboard]

and it starts with its own command:

brozzler-dashboard

After that the dashboard is served at http://localhost:8000/. The documentation points at brozzler-dashboard --help for the configuration options, which is where you would look for the port, the database connection and anything else.

The word rudimentary is the project's own, and it sets the expectation. The extras list for the dashboard contains flask and gunicorn, so it is a conventional Flask application served by gunicorn, not a full monitoring product. What it gives you is visibility into crawl jobs, which is the thing you actually need when a fleet of workers is running and you want to know whether the queue is draining.

The repository also ships an image named Brozzler-Wayback.png, which points at the other half of the workflow: the point of capturing a page with a recording proxy is that it can be replayed later through a Wayback style viewer.

For a crawler whose output is WARC files, a status dashboard is the smallest useful piece of tooling, and this is it.

The browser needs a screen, and the container disables the sandbox

This is the part of brozzler that surprises people, and it is documented in a note of its own.

The browser requires a graphical environment to run. When brozzler runs on a server, that means deploying additional infrastructure, typically X11. Xvnc4 and Xvfb are named as X11 variants suitable for a server, because they do not display anything to a physical screen, and the vagrant configuration in the repository carries an example setup using Xvnc4. The documentation also records a caveat: when last tested, chromium on Xvfb did not support screenshots, so Xvnc4 is preferred at this time.

That is the honest state of a browser-driven crawler. You are not just installing a package, you are running a browser session, and a browser session needs a display even when there is nobody watching it.

The container image makes the trade explicit. It installs chromium into a Python 3.12 bookworm base and then sets a Chrome argument:

FROM ghcr.io/astral-sh/uv:python3.12-bookworm

RUN apt-get update && apt-get install -y --no-install-recommends chromium

ENV BROZZLER_EXTRA_CHROME_ARGS=--no-sandbox

Disabling the Chrome sandbox is what lets Chromium run as root in a container without a display. It also removes one of the browser's own isolation layers, which is a reasonable thing to do for a crawler fetching hostile pages in a disposable container and a bad thing to copy into any other context.

The repository also contains an ansible directory, which suggests the intended production shape is infrastructure as code rather than a hand built box.

Version pins, extras and the packaging

The packaging tells you where the sharp edges are.

The project is at version 1.9.6, with recent releases v1.9.4 and v1.9.5 published on the same day in May 2026 and v1.9.6 in July, and the last push to the repository was on 2026-10-22. The maintainer list on the package manifest names five people, and the author of record is Noah Levitt.

The extras are where the dependencies live. There is yt-dlp for media capture, dashboard for the Flask application, warcprox for the recording proxy at version 2.4.31 or newer, and rethinkdb, and that last one carries a comment worth reading:

rethinkdb = [
  "rethinkdb==2.4.9",
  "doublethink==0.4.9",
  # Needed because of rethinkdb 2.4.9;
  # can be removed when we can upgrade to 2.4.10.post1
  "setuptools>=80.0.0;python_version>='3.12'",
]

So the database client is pinned to an exact version, with a conditional setuptools dependency needed on Python 3.12 and above, and a note that it can be dropped once the project moves to a newer patch release. That is the most concrete maintenance signal in the whole repository: the project is waiting on a RethinkDB release to clean up its own dependency tree.

The core dependencies are ordinary, with PyYAML, requests, websocket-client, pillow, urlcanon, cerberus for validation, jinja2, cryptography, python-magic, prometheus-client and structlog for structured logging. Prometheus metrics and structured logs are built in, which matters for a service with a fleet of workers.

The licence is Apache-2.0, copyright 2015 to 2025 Internet Archive, which is the most permissive option available for this kind of institutional software, and the repository keeps a license.txt alongside the pyproject declaration.

Editorial conclusion

brozzler suits institutions that archive the live web and need pages captured as a browser sees them, including content that only appears after JavaScript runs, and that already run warcprox and RethinkDB as services. It is the wrong choice for a one-off crawl of a small, static site, where an HTTP crawler is faster and simpler, and it is the wrong choice if you cannot provide a display for the browser, since the container image disables the Chrome sandbox to work around exactly that. Verify three things before you deploy it: that the version of RethinkDB you have matches the pin the project carries, that warcprox instances are registered in the same service table the workers use, and how the archived output of your particular pages looks in the replay viewer before you point a full crawl at them.

Frequently asked questions

Official sources

1. internetarchive/brozzler on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases