# pyperf: running Python benchmarks that survive someone asking for the numbers again

> pyperf is a Python toolkit to write, run and analyze benchmarks, built around automatic calibration, worker processes and statistical comparison of saved result files rather than a single timing number.

**psf/pyperf** — Toolkit to run Python benchmarks

- Repository: https://github.com/psf/pyperf
- Website: http://pyperf.readthedocs.io/
- Stars: 968 · Forks: 112
- Language: Python
- License: MIT
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/psf-pyperf

## A timing number is not a benchmark result

The stated problem is not that Python is slow to measure, it is that a single measurement tells you almost nothing. A raw timing depends on what else the machine was doing, on how many times the code ran, on whether the interpreter had warmed up. pyperf is a toolkit to write, run and analyze benchmarks, and the feature list is built around removing that noise: automatic calibration of a benchmark to a time budget, spawning multiple worker processes, computing the mean and standard deviation, detecting when a result seems unstable, storing results in JSON format, and support for multiple units including seconds, bytes and integers. The audience is people who need to defend a number. That includes CPython contributors tracking performance between releases, and any team whose pull requests are judged on whether they made something slower.

## Calibration decides how much work happens

The design detail that separates pyperf from a timing loop is that the amount of work is not fixed by hand. The toolkit calibrates a benchmark against a time budget, and it reports what it decided. The statistics output breaks the run down into layers: the number of calibration runs, the number of runs that produced values, the total number of runs, warmups per run, values per run and loop iterations per value. Each layer answers a different question about reliability. Warmups exist because the first iterations of any Python code are not representative, values per run protect against a single slow moment, and loop iterations amortise the timer overhead when the statement under test is fast. Because the calibration is automatic, two people on different machines can still produce comparable structures, which is what makes the saved files worth comparing.

## Your first run, one command at a time

Installation is a single step:

```shell
python3 -m pip install pyperf
```

The quickest way to get a result is the `timeit` command, which runs a statement and writes the outcome into a JSON file:

```shell
$ python3 -m pyperf timeit '[1,2]*1000' -o bench.json
```

The same work written as a script gives you more control, because you can name the benchmark and declare its setup:

```python
    #!/usr/bin/env python3
    import pyperf

    runner = pyperf.Runner()
    runner.timeit(name="sort a sorted list",
                  stmt="sorted(s, key=f)",
                  setup="f = lambda x: x; s = list(range(1000))")
```

The script runs with the same output option:

```shell
$ python3 bench.py -o bench.json
```

Note what the setup line does: it builds the list once, outside the timed statement, so the measurement covers the call rather than the list construction.

## Reading the results back out of the JSON file

Analysis is a separate command, and it reads the file rather than rerunning anything:

```shell
$ python3 -m pyperf stats telco.json
```

The output is dense by design. It gives the total duration and the start and end dates, the raw minimum and maximum, then the run structure described earlier, then the distribution: minimum, median with MAD, mean with standard deviation, maximum, the 0th, 5th, 25th, 50th, 75th, 95th and 100th percentiles, and a count of outliers relative to the observed range. Two different spread measures are worth noticing. The mean with standard deviation is what most people quote, while the median with MAD is the more resistant pair, since one slow run drags a mean further than it drags a median. An outlier count of zero is the signal that a run was quiet enough to trust.

## Deciding whether a difference is real

This is the command that changes how a benchmark argument gets settled. `pyperf compare_to` tests whether a difference is significant, and it supports comparison between multiple benchmark suites made of multiple benchmarks:

```shell
$ python3 -m pyperf compare_to --table mult_list_py36.json mult_list_py37.json mult_list_py38.json
```

The table output expresses results as a ratio against a reference suite, so a cell reads as a factor faster or slower, and a cell can also read as not significant. That last case is the point. Two suites that differ by a few percent routinely differ by a few percent in whichever direction the noise went, and only a significance test distinguishes that from a regression worth investigating. A geometric mean row summarises the suite as a whole. The alternative to all of this machinery is a timing loop in a throwaway script that prints one number, and the cost of that shortcut is exactly the calibration, worker isolation and significance testing described above.

## Tuning the host and recording what the host was

pyperf acknowledges that the machine is part of the measurement. The `pyperf system tune` command tunes your system to run stable benchmarks, which is aimed at laptops and desktops where background activity and frequency scaling add noise. Alongside it, metadata is collected automatically on the computer and on the benchmark itself, displayed with the `pyperf metadata` command or gathered on demand with `pyperf collect_metadata`. That record matters when a result is disputed, because it documents the hardware and the environment rather than leaving them to be reconstructed later. Memory is a separate axis: the `--track-memory` and `--tracemalloc` options track the memory usage of a benchmark, so a change that trades memory for speed still shows up in the same JSON file rather than being argued about separately.

## Release cadence, versioning and the licence

pyperf lives under the Python Software Foundation, ships under the MIT licence and is covered by the PSF Code of Conduct. The version number is dynamic in `pyproject.toml`, which points at `pyperf/__init__.py`, and the release procedure is written down in the same file: pull with rebase, update the version in `pyperf/__init__.py` and `doc/conf.py`, set the release date in `doc/changelog.rst`, run the tests with `tox --parallel auto`, then publish from the GitHub release tab and watch the publish workflow. The tagged releases are 2.10.0 on 2026-02-07, 2.9.0 on 2025-03-04 and 2.8.1 on 2024-11-15, and the last push was on 2026-10-01. The gap between the last two tagged releases is worth noticing before you plan around a cadence, and the project classifies itself as Production/Stable in its packaging metadata.

## Conclusion

Adopt pyperf when benchmark numbers get quoted in reviews and pull requests, since the JSON result files, the calibration and the significance test are what make two runs comparable. Do not adopt it for a one-off timing, where the overhead of calibration and worker processes buys you nothing. Verify first that the machine is quiet during a run, because `pyperf system tune` and the metadata collection exist precisely because a busy host turns a valid comparison into noise, and check whether your CI runner is dedicated before trusting a result collected on a shared one.

## FAQ

### How do I install pyperf?

With `python3 -m pip install pyperf`. The documentation is at pyperf.readthedocs.io, and the source and issue tracker live in the psf/pyperf repository on GitHub.

### How does pyperf decide how many times to run a benchmark?

It calibrates the benchmark automatically against a time budget and spawns multiple worker processes. The statistics output reports the calibration runs, the warmups per run, the values per run and the loop iterations per value, so the structure of a run is visible in the result.

### How do I tell whether a performance difference is significant?

Use `pyperf compare_to`, which tests whether a difference is significant and supports comparison between multiple benchmark suites. Its table output marks results as faster or slower relative to a reference, and marks differences that are not significant.

### Can pyperf measure memory usage as well as time?

Yes. The `--track-memory` and `--tracemalloc` options track the memory usage of a benchmark, and the results are stored alongside timing data in the same JSON result file.

## Sources

- [License: MIT](https://github.com/psf/pyperf/blob/main/LICENSE)
- [Project website](http://pyperf.readthedocs.io/)
- [psf/pyperf on GitHub](https://github.com/psf/pyperf)
- [README](https://github.com/psf/pyperf/blob/main/README.md)
- [Releases](https://github.com/psf/pyperf/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/psf-pyperf