Browsertime: the browser measurement engine behind sitespeed.io

A measurement engine, not a reporting tool

Browsertime is an open-source Node.js tool designed for engineers who are building their own performance measurement tools. It serves as the core engine of projects like sitespeed.io, and it is described as a trusted tool used by Mozilla to measure the performance of Firefox.

That framing is the most important thing to understand before you install it. Browsertime is not a site speed product with a dashboard and a subscription. It is the layer that actually drives the browser, collects the numbers and writes the files, and sitespeed.io is one consumer of it. If you want to build a performance pipeline, a regression check, or a custom measurement harness, this is the part you would otherwise have to write yourself, and the fact that Mozilla measures Firefox with it is the reference point for how much of the browser's own performance work it can carry.

The key features are listed in four groups. Performance testing covers page load times, resource loading and user interactions. Deep metrics cover page load times, visual metrics for when things are painted on the screen, and largest contentful paint among others. Scripting lets you customise tests and user interactions with JavaScript. And Docker support is for integrating into CI and containerised environments.

The repository layout is consistent with that. There is a lib/ directory for the implementation, a bin/ directory for the command line entry point, browserscripts/ for the code injected into the page, visualmetrics/ for the visual metric computation, types/ and a scripting.d.ts for the public type surface, an index.js as the module entry, and a docker/ directory. There is also a requirements.txt, which tells you something about the architecture before you read a line of the code.

Installation is a global npm package or a browser-laden image

There are two installation routes and they differ more than usual.

The Node.js route installs the CLI globally and runs it against a URL:

npm install -g browsertime
browsertime https://example.com

The Docker route runs the published image:

docker run --rm -v "$(pwd)":/browsertime sitespeedio/browsertime https://www.sitespeed.io/

The difference in what you get is substantial, and it is visible in the Dockerfile. The image is built FROM sitespeedio/webbrowsers:chrome-153.0-firefox-155.0-edge-153.0, so it already contains three browsers at fixed versions. It installs libnss3-tools, net-tools and tcpdump, and it prepares an NSS certificate database, which is the step WebPageReplay needs to install a local root certificate. It copies the WebPageReplay binary and its certificate and key, plus a deterministic.js script. It sets three environment variables that shape the whole run: BROWSERTIME_XVFB=true, BROWSERTIME_CONNECTIVITY__ENGINE=external and BROWSERTIME_DOCKER=true.

The /browsertime directory is declared as a volume, which is where the results land and how they get back to your host. The entrypoint is a start script rather than the CLI itself, and the image allows a set of networking commands to run under sudo without a password, specifically the ones the throttle package needs to shape traffic.

The Node route leaves all of that to you. You get the tool and you get the browsers you already have.

One detail to note if you plan to run the npm version: the manifest declares the engine requirement as node >=22.0.0, and the dependency list pins browser drivers to specific versions, chromedriver at 153.0.8010, edgedriver at 153.0.4234 and geckodriver at 0.37.1. Those pins have to line up with the browsers you have installed, and the Docker route sidesteps that by shipping matched browsers.

Running it, and the options that matter

The basic usage is one flag and a URL:

browsertime https://www.example.com --browser chrome

The number of runs is the first thing to set, because a single run is noise:

browsertime -n 5 https://www.example.com

The browser is selectable:

browsertime --browser firefox https://www.example.com

And custom metrics are gathered by evaluating a script against the page, which is the mechanism for pulling a value out of your own application:

browsertime --script "return {'myMetric': window.myApp.customMetric}" https://www.example.com

That last one is how you extend the metric set beyond what the tool collects by default, and it is the feature that makes Browsertime an engine rather than a tool: your metric comes from your application, evaluated in the page, at the moment the measurement is taken.

The full set of options is available from the command itself, and the documentation says to run browsertime --help for the configuration options. The library is also usable programmatically, and the manifest exposes a single root export with types pointing at scripting.d.ts, so a tool that embeds the engine gets the same script interface.

Connectivity is a separate concern and is handled by a dedicated package rather than by Browsertime itself. The recommendation for making the connection slower, in order to catch regressions more easily, is to set up a network bridge in Docker or to use the Throttle engine, and the Docker image is preconfigured to use the external engine.

WebPageReplay, and what recording changes

The Docker container includes WebPageReplay, and the workflow it enables is the most interesting thing in the documentation.

The purpose is to replay your page locally, getting rid of server latency, which makes it easier to find front end regressions. The sequence is explicit. The start script starts WebPageReplay in record mode. Then Browsertime accesses the URL you choose, once, so that it is recorded. WebPageReplay is then shut down. WebPageReplay is started again in replay mode. Browsertime accesses the URL as many times as you choose. And then replay mode is shut down too.

So the recording happens once and the measurement happens many times against a local copy. That is a methodological change, and it is the point: when the server is removed from the loop, the variance in your numbers comes from the front end and your network conditions rather than from the origin's response time.

The latency is configurable through a Docker environment variable, and REPLAY turns the replay functionality on. The Chrome invocation:

docker run --cap-add=NET_ADMIN --rm -v "$(pwd)":/browsertime -e REPLAY=true -e LATENCY=100 sitespeedio/browsertime:20.0.0 https://en.wikipedia.org/wiki/Barack_Obama

And the Firefox equivalent, which also shows how the browser flag and the number of runs are combined:

docker run --cap-add=NET_ADMIN --rm -v "$(pwd)":/browsertime -e REPLAY=true -e LATENCY=100 sitespeedio/browsertime:20.0.0 -b firefox -n 11 --firefox.acceptInsecureCerts true https://en.wikipedia.org/wiki/Barack_Obama

Note the NET_ADMIN capability, which is what the throttle engine needs to shape traffic, and the acceptInsecureCerts flag for Firefox, which is there because the replay proxy presents a local certificate.

Mobile testing, video and the visual metrics stack

Browsertime supports Chrome and Firefox on Android, collecting SpeedIndex, HAR and video. Two prerequisites are named: you need to install adb, and you need to prepare your phone, with documentation for both.

The mobile invocation on a device looks like this:

$ browsertime --chrome.android.package com.android.chrome https://www.sitespeed.io --video --visualMetrics

The chrome.android.package option names the package to drive, which is how you pick between the installed browsers on the handset. Setting connectivity on a phone needs something like gnirehtet, because the traffic shaping has to happen on the device rather than in a container.

The Android support is also visible in the packaging. The manifest includes @devicefarmer/adbkit for talking to the device, and the optional dependency usb-power-profiling is there for power measurement over USB. The Dockerfile adds an insecure shared adb key into a root-owned .android directory, with a comment explaining that this avoids having to click an OK button during the build, which is the kind of detail that only appears in a container that is meant to run unattended.

The visual metrics side is Python, which is the surprise in an otherwise Node.js tool. The requirements file lists Numpy and Pillow as core, and then two optional packages with a comment explaining exactly when each is needed: pyssim only for --visualMetricsPerceptual, and OpenCV-Python only for --visualMetricsContentful. The published files include visualmetrics/visualmetrics-portable.py, so the Python component is shipped alongside the Node code rather than being a separate service.

That split is the reason the visual metrics are a separate concern at all. Perceptual similarity and contentful comparison are image comparisons, and doing them properly needs an image library and a similarity metric rather than a JavaScript implementation.

Where Browsertime is the wrong tool

The first limitation is that it measures a page, not an API. Everything here happens in a real browser against a real URL, which is the right way to measure a user experience and the wrong way to measure a service's throughput. If you want request rate, latency distribution under load or error rates, you want a load testing tool, and no amount of browser automation will give you that.

The second limitation is the environment cost. The npm route needs Node 22 and a matching set of browser drivers, and it will break in a way that is annoying to diagnose when a driver version and an installed browser version drift apart. The Docker route avoids the version problem and replaces it with an image that carries three browsers, a WebPageReplay binary, Python imaging libraries, tcpdump and a set of sudo permissions for network shaping. Either way you are taking on a fairly heavy footprint for a measurement tool, and the container needs NET_ADMIN or privileged mode for the throttling and replay features to work at all, which is a constraint in a locked-down CI environment.

The third limitation is that the output is files rather than insight. The documentation talks about collecting HAR, video and metrics, and the repository has a jsdoc/ directory and a types/ directory, but the result of a run is data that something else has to interpret. If your goal is a page speed score in a dashboard, sitespeed.io is the tool that already does that on top of this engine, and writing your own reporting layer on Browsertime is work you have chosen rather than work you had.

The real alternative for most teams is therefore sitespeed.io, which is built on Browsertime and adds the configuration, the reporting and the browsing. The difference in approach is that sitespeed.io is a product for measuring sites, while Browsertime is a library and a CLI for people who are building their own measurement tooling, which is the situation the project describes itself as serving. The other alternative, and the one for smaller jobs, is the browser's own devtools performance panel: no installation, no driver management, no container, and a recording you read by hand. That is fine for investigating one slow page and useless for a regression suite, which is exactly the line Browsertime sits on the other side of.

Editorial conclusion

Adopt Browsertime if you are building your own performance measurement tooling and want a maintained engine that handles browser drivers, HAR export, visual metrics and video, and if you can live with a Node 22 requirement and a dependency set that tracks specific browser driver versions. Do not adopt it as a ready-made performance dashboard, because the output is files and metrics rather than a report, and do not use it for synthetic API benchmarking, since it measures a real page load in a real browser. Before you build on it, check that the pinned chromedriver, edgedriver and geckodriver versions match the browsers installed on your machine, and decide whether you want the WebPageReplay workflow, because recording and replaying changes what the numbers mean.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. sitespeedio/browsertime on GitHub