OpenWPM: Firefox and Selenium for web privacy measurement

OpenWPM's measurement scope and intended research users

OpenWPM targets web privacy studies that need to collect data across thousands or millions of websites. Its intended users are measurement researchers who need repeatable browser automation rather than a general-purpose scraping library. The repository identifies Python as its primary language, while its browser behavior depends on Firefox, Selenium, an instrumentation extension, and a Python configuration layer.

The important distinction is the measurement context. OpenWPM is built to observe browser activity during controlled visits, with hooks placed where researchers can collect relevant data. The README links the configuration documentation as the authoritative description of available collection instruments. That omission matters. A team should not infer that OpenWPM captures cookies, storage, network behavior, or script activity by default merely because it is a privacy framework. The exact collection scope must be checked against `openwpm/config.py`, the selected instrument configuration, and `docs/Configuration.md`.

The project also supports a demo based on the Tranco top sites list through `demo.py --tranco`. The README warns that this is a real top-sites list, so it can include NSFW websites. That warning makes the option unsuitable for an unattended first run unless the operator has considered content exposure and the research data handling required by the environment.

How OpenWPM drives Firefox and collects measurements

OpenWPM uses Selenium to automate an unbranded Firefox installation. The architecture is divided among several repository components. Python packages under `openwpm/` provide configuration, browser management, commands, instrumentation integration, logging, and storage code. The `Extension/` directory contains the instrumentation extension source, which `install.sh` builds. A measurement entry point such as `demo.py` can be replaced or complemented by a larger crawler implemented through repository modules.

The browser parameters determine how Firefox is launched and controlled. The README specifically calls out `display_mode` for headless environments, with `"headless"` given as the required value there. If every spawn fails with a message saying Firefox exited before Selenium could connect, the documented first checks are the Firefox and Selenium versions. The versions should match the requirements in `install.sh` and `environment.yaml`.

A crawler process creates a managed browser session, visits target sites, and invokes configured instrumentation through the OpenWPM layer. Results can be retained locally or sent to remote storage, according to the README's storage section. The repository also contains `schemas/`, which suggests structured configuration inputs, and Node utilities in `package.json` can render schema documentation. The README does not define the output schema, event identifiers, storage protocol, retry policy, or serialization format. Those details cannot safely be inferred and should be verified before using the output in a published study.

Installing OpenWPM in its supported Linux environment

The documented target is Ubuntu 24.04, where the project says its tests run through GitHub Actions. The Docker image uses Ubuntu 22.04. Conda is the main prerequisite. On macOS, the README says installation may work, but macOS CI is absent and Firefox with xvfb is explicitly untested. Windows is unsupported.

From the repository root, run the supplied installer. It creates the conda environment, installs unbranded Firefox, and builds the instrumentation extension. It also overrides an existing conda environment named `openwpm`, which is a significant setup constraint.

./install.sh

After a successful installation, activate the generated environment before running the demo or any other entry point.

conda activate openwpm

On macOS, the README identifies the Xcode command-line tools as the route to obtaining `make` and `gcc`, which may be needed to build the extension. Because macOS is not covered by CI, an installation succeeding there should not be treated as equivalent to the documented Ubuntu environment. Ubuntu installations also need the build and browser dependencies represented in the repository's `Dockerfile` if the host lacks them.

Running the first OpenWPM demo and a Tranco crawl

The repository provides `demo.py` as the first-use example. It uses defaults from `openwpm/config.py::ManagerParams` and `openwpm/config.py::BrowserParams`, except for changes made inside the demo. Running the script starts with that configuration rather than an empty framework.

python demo.py

The README does not document a specific success message or output file for this command, so success should be confirmed through the demo's actual output and the configured storage behavior. The Docker image uses the same command as its default container command, which confirms the entry point but does not define the measurements written.

To use the optional Tranco top sites input, add the documented flag. This changes the target source rather than the basic OpenWPM architecture.

demo.py --tranco

Because the list can contain NSFW websites, this command should not be treated as a harmless sample dataset. For a study, the operator must also select an appropriate instrument set, storage destination, browser settings, and concurrency level. The README says to consult `docs/Using_OpenWPM.md` for the full tutorial and `docs/Configuration.md#Instruments` for collection methods. Those files, rather than the short demo description, are the right starting points for defining a production measurement.

Storage, concurrency, and crawl failure constraints

OpenWPM documents both local and remote storage, so a deployment must decide where measurement records go before a large crawl begins. The README does not specify the remote backend, authentication model, retention policy, or recovery process. That is a meaningful boundary for research data. A local result directory and a remote object store do not imply the same failure behavior, and the documentation does not confirm transactionality or deduplication.

Browser concurrency is also a practical constraint. The troubleshooting guidance says random Firefox startup failures during crawls can result from an overtaxed system, including memory or CPU pressure. The documented response is to lower the number of concurrent browsers. This is more informative than a generic warning because it connects crawl concurrency directly to browser reliability.

On remote or otherwise headless systems, every browser must use `display_mode` set to `"headless"` before launch. Missing that setting can cause Firefox startup to fail before Selenium connects. For environment contamination, the README documents `PYTHONNOUSERSITE=True` as a way to exclude user-installed Python packages. Prepending it to the demo command addresses the documented package-resolution conflict without changing the repository configuration.

PYTHONNOUSERSITE=True python demo.py

The README does not document rollback for a partially completed crawl, resume semantics, or a recovery point after worker failure. Large studies should therefore treat crawl scheduling and output completeness as explicit protocol concerns rather than assuming that browser restart behavior equals dataset recovery.

OpenWPM maintenance, release work, and licensing boundaries

The repository is not archived, and its last push was on 2026-10-13. Recent releases include `v0.38.0` on 2026-10-04, `v0.37.0` on 2026-09-06, and `v0.36.0` on 2026-08-24. This supports treating the current development line as recently changed. It does not mean that every experimental configuration or non-Ubuntu platform has the same support boundary.

Upgrades have a real environment cost. Re-running `install.sh` installs the conda environment, Firefox, and extension, but it can override the existing environment named `openwpm`. Researchers using a pinned study environment should preserve that environment and compare `install.sh`, `environment.yaml`, `VERSION`, and the release notes before replacing it. The repository also contains `CHANGELOG.md`, `CONTRIBUTING.md`, tests, and CI configuration, but the README gives no upgrade procedure or backward-compatibility promise.

The repository-level license classification is custom and points to the `LICENSE` file. Separately, `package.json` declares `"license": "MPL-2.0"` for NodeJS utilities. Those statements may cover different components, so they should not be collapsed into one conclusion. An organization must inspect `LICENSE` and any component-specific notices itself. This review does not provide a legal interpretation or assurance that every bundled Firefox, extension, or dependency asset has the same terms.

Where OpenWPM differs from a conventional HTTP crawler

A conventional HTTP crawler generally retrieves pages and follows links through an HTTP client. OpenWPM instead automates Firefox through Selenium, which means each target is exercised in a browser environment capable of executing client-side web technology. That approach is necessary when the research question depends on browser-mediated privacy behavior rather than only server responses or static HTML.

The trade-off is operational weight. OpenWPM requires a conda environment, a compatible Firefox build, Selenium, an extension build, and enough host capacity for concurrent browser processes. HTTP clients are generally easier to run in constrained environments and usually make transport-level failures more visible. OpenWPM instead adds browser startup, rendering, extension, storage, and system-load failure modes.

OpenWPM is the wrong tool when the study only needs ordinary page retrieval, does not need browser execution, or cannot support a Firefox-managed environment. It is also unsuitable as-is on Windows. A lightweight HTTP crawler is a better fit for URL retrieval and link traversal, while OpenWPM is appropriate when the measurement protocol specifically depends on a scripted Firefox session and configurable instrumentation. The decision rests on the observation target, not on a general claim that one crawler is more capable than the other.

Editorial conclusion

Adopt OpenWPM when a research team needs reproducible Firefox crawls and configurable privacy data collection at scale. Do not choose it for Windows work, macOS deployments requiring tested CI coverage, or studies that require immediate rollback and disaster recovery guarantees that the documentation does not describe. Before adoption, verify that the selected instruments, local or remote storage path, concurrency level, and custom LICENSE terms match the study protocol.

Frequently asked questions

Official sources

1. Issues
2. openwpm/OpenWPM on GitHub
3. Project website
4. README
5. Releases