# PerfKitBenchmarker: a cloud VM provisioning harness for standardized, cross-provider benchmark comparison

> PerfKitBenchmarker (PKB) provisions cloud VMs, runs over 30 wrapped benchmark tools, and tears down the environment. Benchmark defaults are fixed to keep numbers comparable across providers, and a consent flag is required on every run.

**GoogleCloudPlatform/PerfKitBenchmarker** — PerfKit Benchmarker (PKB) contains a set of benchmarks to measure and compare cloud offerings. The benchmarks use default settings to reflect what most users will see. PerfKit Benchmarker is licensed under the Apache 2 license terms. Please make sure to read, understand and agree to the terms of the LICENSE and CONTRIBUTING files before proceeding.

- Repository: https://github.com/GoogleCloudPlatform/PerfKitBenchmarker
- Website: https://googlecloudplatform.github.io/PerfKitBenchmarker/
- Stars: 2,014 · Forks: 563
- Language: Python
- License: Apache-2.0
- Published: 2026-10-11 · Updated: 2026-10-11 · Language: en
- Canonical page: https://hysenlabs.com/projects/googlecloudplatform-perfkitbenchmarker

## Cloud VM provisioning and workload orchestration, not a standalone measurement engine

PerfKitBenchmarker is an orchestration tool. It provisions virtual machines on a cloud provider, installs benchmark software, runs the workloads, and tears down the environment when the run completes. The benchmarks themselves are pre-existing tools: fio, iperf, memtier_benchmark, netperf, and others listed in the README's benchmark table. PKB wraps them with automated provisioning, configuration, and result collection. The project has no measurement engine of its own; results come from the underlying tools, and PKB's contribution is consistency in how those tools are invoked across providers.

The repository is organized around a Python package under perfkitbenchmarker/, a main entry point at pkb.py, and a tutorials/ directory. The README links to two starting paths: a beginner walkthrough using GCP's Cloud Shell with netperf benchmarks, and a Docker-based path. Extended documentation lives in the project's GitHub wiki, which covers governing rules, design documents, Tech Talks, community meeting decks and notes, and a FAQ that the README does not reproduce.

For single-provider use cases where the goal is performance maximization rather than provider comparison, native load-testing tools avoid the overhead of PKB's VM provisioning layer. PKB's value is the cross-provider comparison, not raw performance extraction from a single environment.

## Benchmark defaults stay fixed to keep cross-provider numbers comparable

PKB's governing rule is stated in the README: benchmark defaults are not tuned for any particular platform or instance type. The settings are recommended for consistency across services. PKB changes a benchmark's configuration only when a common practice exists for that workload category; setting the buffer pool size on a database benchmark is the README's example of a justified exception.

This constraint has a direct consequence for teams wanting to maximize throughput from a specific instance type. A run tuned for high-memory instances will not produce numbers comparable to a run on a different instance type using the same benchmark at default settings. PKB is built for the comparison case.

The governing rules for when a default may change are documented in the project's GitHub wiki rather than in the README. Any deviation from defaults requires that change to be recorded under those rules rather than applied silently. The README presents this as a deliberate design choice, and the wiki is the right place to look when a benchmark's defaults need reconsideration.

## Python dependencies, Docker, and the consent flag PKB requires before any benchmark run

PKB is a Python project with all dependencies listed in requirements.txt at the repository root. Install them with:

```bash
pip install -r requirements.txt
```

The setup.py lists a minimal install_requires of absl-py, jinja2>=2.7, and setuptools. The full requirements.txt adds numpy>=1.16.5, pandas>=1.1.5, PyYAML>=5.4.1, scipy, jinja2>=2.10.2, google-cloud-monitoring>=2.0.0, and pywinrm, among others. The pywinrm package is a Windows Remote Management library; its presence in the requirements signals that PKB can benchmark Windows-based cloud VMs, not only Linux instances.

A Docker path is also available. The Dockerfile sets Python 3.9 as the base image argument, creates /pkb as the working directory, and installs requirements.txt followed by requirements-testing.txt. The CMD in the Dockerfile runs the test suite rather than a benchmark; actual benchmark runs go through the pkb.py entry point.

Before any benchmark run, PKB requires the --accept-licenses flag. The README states that all runs must include it. Benchmark tools wrapped by PKB each operate under their own separate terms, and PKB provides no mechanism to check those terms on the user's behalf; the flag confirms the operator has reviewed and agreed to them.

## The benchmark table covers 30+ tools, one deprecated MongoDB entry, and a broad range of workload types

PKB's README benchmark table lists over 30 tools. The workload categories span storage benchmarks (fio, bonnie++), network throughput (iperf, netperf), memory (multichase), database workloads (cassandra_ycsb, sysbench_oltp, pgbench), and GPU measurements (gpu_pcie_bandwidth). Each entry in the table names the tool and its terms, and PKB asks users to review those terms before running.

A detail in the table that is easy to miss: the `mongodb` entry is marked 'Deprecated' in the README, but `mongodb_ycsb`, which also targets MongoDB through the YCSB workload driver, carries no deprecation label. Both entries appear in the same section of the benchmark table, but only `mongodb` carries a deprecation label. A team checking the README for MongoDB benchmark coverage would find two entries with different statuses sitting side by side.

PerfKit Benchmarker is licensed under Apache 2.0, and the last push was on 2026-10-06. The benchmarks it wraps are governed by their own separate terms, which range from permissive open-source to EEMBC's proprietary EEMBC license for coremark; the --accept-licenses flag exists because those terms vary significantly across the set.

Some benchmarks also require Java. The openjdk-7-jre dependency must be agreed to separately by any user running a Java-dependent benchmark.

## SPEC CPU2006 is the one benchmark PKB cannot automate: the ISO must be purchased and placed manually

Every benchmark in PKB's set can be provisioned automatically, with one exception. SPEC CPU2006 requires users to purchase access from spec.org, agree to SPEC's terms and conditions, and manually download cpu2006-1.2.iso from the SPEC website.

Once obtained, the ISO goes at perfkitbenchmarker/data/cpu2006-1.2.iso. A runspec configuration file is also required; the README gives linux64-x64-gcc47.cfg as an example filename, placed at perfkitbenchmarker/data/linux64-x64-gcc47.cfg.

An alternative path avoids the ISO entirely: extract the ISO's contents into a directory named cpu2006, run cpu2006/install.sh, copy the cfg file into cpu2006/config, create a tar file of the cpu2006 directory, and place it at perfkitbenchmarker/data/cpu2006v1.2.tgz. PKB uses the tar file if present; otherwise it searches for the ISO and cfg files.

This manual setup means SPEC CPU2006 cannot be added to an automated benchmark pipeline without first completing the SPEC purchase and file placement steps on each machine that runs PKB. For teams that need SPEC results as part of a standard cross-cloud comparison, those steps are prerequisites that PKB cannot handle on its own.

## No pip package path in the README: installation goes through cloning and requirements.txt

PKB is not described in the README as distributable through a PyPI package. The setup.py defines the package name as 'perfkitbenchmarker' and registers pkb.py as an installed script, but the documented path is cloning the repository and running pip against requirements.txt locally. Teams that automate PKB deployment in CI pipelines or on cloud instances need to include the clone step in their provisioning scripts rather than relying on a standard pip install command.

The repository root holds two change log files: CHANGES.md and CHANGES.next.md. The README does not describe or link to either file; their scope and relationship to each other are not documented outside the files themselves.

For teams running PKB over time, one requirements.txt constraint warrants attention: setuptools>=40.3.0,<81. The upper bound means that upgrading the Python environment to setuptools 81 or above would break the install until requirements.txt is updated. Any broad Python environment update across machines running PKB should verify that setuptools remains within that pinned range before proceeding with the upgrade.

## When PKB is the wrong tool: non-cloud deployments, cost-unaware runs, and approval-gated environments

PKB provisions VMs through vendor-provided command line tools. This creates a hard dependency on a cloud provider that exposes those tools. The README describes no documented path for on-premise hardware, private clouds, or environments where cloud VM provisioning requires approval workflows that operate outside the CLI. If your infrastructure does not match the cloud-provision model, PKB's core automation does not apply.

Every benchmark run incurs cloud costs because VMs are provisioned on the target provider for the duration of the run. PKB has no built-in cost estimation or budget-limiting mechanism documented in the README. Teams operating within per-run cost limits must track expenses through the cloud provider's billing tooling rather than through PKB itself, which provides no budget-capping mechanism.

The beginner walkthrough tutorial uses GCP's Cloud Shell, which requires an active GCP account and project. Teams running on a different provider need to map the tutorial steps to their own provider's CLI environment. The governing rules for any configuration exception are in the wiki rather than the README, which means teams considering a documented deviation from defaults must locate and read those wiki pages before any non-standard run.

## Conclusion

Use PKB when you need reproducible, provider-neutral numbers from unchanged benchmark defaults: that consistency is the tool's defining constraint, and it makes cross-provider comparisons meaningful. The same constraint rules out tuned results for a specific instance type; the two goals conflict. Before running, verify that each wrapped benchmark's terms are acceptable to your organization, since the tools differ widely and PKB's consent flag is the only guardrail PKB provides. For Windows VM benchmarks, check pywinrm is configured in your environment; the requirements.txt includes it but no Windows setup guide appears in the README.

## FAQ

### How do I install PerfKitBenchmarker?

Clone the repository and install dependencies with `pip install -r requirements.txt`. No PyPI package path is described in the README. A Docker option is also available: the Dockerfile sets Python 3.9 and installs requirements.txt automatically.

### What is the --accept-licenses flag in PerfKitBenchmarker?

PKB wraps benchmark tools that each operate under their own separate terms. The --accept-licenses flag confirms that the operator has read and agreed to each tool's terms. The README states this flag is required on every run.

### Can PerfKitBenchmarker benchmark Windows cloud VMs?

The requirements.txt includes pywinrm, a Windows Remote Management library, which indicates support for Windows cloud VMs. The README does not include a Windows-specific setup or configuration guide.

### Can PerfKitBenchmarker benchmark non-cloud infrastructure?

PKB provisions VMs through vendor-provided cloud CLI tools. The README describes no documented path for on-premise hardware or private clouds that do not expose the same CLI interface.

### How do I run SPEC CPU2006 with PerfKitBenchmarker?

SPEC CPU2006 cannot be automated by PKB. You must purchase access from spec.org, download cpu2006-1.2.iso manually, and place it at perfkitbenchmarker/data/cpu2006-1.2.iso. A runspec configuration file is also required before PKB can invoke the benchmark.

## Sources

- [GoogleCloudPlatform/PerfKitBenchmarker on GitHub](https://github.com/GoogleCloudPlatform/PerfKitBenchmarker)
- [License: Apache-2.0](https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/LICENSE)
- [Project website](https://googlecloudplatform.github.io/PerfKitBenchmarker/)
- [README](https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/README.md)
- [Releases](https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/googlecloudplatform-perfkitbenchmarker