pytest-benchmark turns a pytest fixture into a calibrated micro benchmark

Rounds are sized from a calibration of your timer

The plugin describes itself as a pytest fixture for benchmarking code, and the second half of that sentence is the interesting part: it groups the tests into rounds that are calibrated to the chosen timer.

That is a response to a specific problem with naive micro benchmarks. If you call a function once and time it, you measure the noise. If you call it a fixed number of times, the result depends on how long that number takes relative to the clock's resolution. The calibration step measures the timer itself, works out how many iterations are needed to get above the noise floor, and then divides the work into rounds so that each round produces a usable sample. The documentation has a separate page on calibration for the details, and the FAQ page covers the questions that come up around it.

The practical consequence is that a benchmark written as a pytest test is a statistical measurement rather than a stopwatch reading, and the summary table it prints reports the distribution of the rounds rather than a single number.

The timing code and ideas are credited to vstinner's benchmark.py, and the plugin is free software under the BSD 2-Clause licence, which is what the project metadata records and the LICENSE file holds.

The fixture is a callable, and you call it with the function under test

The API is one fixture. This plugin provides a benchmark fixture, and the fixture is a callable object that will benchmark any function passed to it. The example in the README shows a function that needs benchmarking and a test that uses the fixture.

def something(duration=0.000001):
    """
    Function that needs some serious benchmarking.
    """
    time.sleep(duration)
    # You may return anything you want, like the result of a computation
    return 123

The test itself takes the fixture as an argument and calls it with the function, then keeps the return value so the test can also assert on correctness.

def test_my_stuff(benchmark):
    # benchmark something
    result = benchmark(something)

    # Extra code, to verify that the run completed correctly.
    # Sometimes you may want to check the result, fast functions
    # are no good if they return incorrect results :-)
    assert result == 123

The comment about asserting on the result is the point of the design rather than an aside. A benchmark that measures a function returning the wrong answer is a benchmark of nothing, and because the fixture is a normal pytest fixture, the correctness assertion lives in the same test as the measurement.

The function can also take extra arguments, positionally with benchmark(time.sleep, 0.02), or as keyword arguments with benchmark(time.sleep, duration=0.02). Both forms are in the README.

One function per test, and a warning about the decorator pattern

The README's prologue is a list of constraints, and they are the real API documentation.

The plugin benchmarks functions and only that. If you want to measure a block of code or a whole program, you need to write a wrapper function. In a test you can only benchmark one function, and if you want to benchmark many functions you write more tests or use pytest's parametrization. And to run the benchmarks you simply use pytest to run your tests, because the plugin automatically does the benchmarking and generates a result table, with pytest --help giving more details.

The one function per test rule has a reason behind it. A fixture call is a measurement session, and the plugin's calibration and round logic assume one measurement per test. Parametrization is the way to get many measurements, because each parameter set becomes its own test.

The third note is a warning about a pattern the README says is seen in the wild but is not recommended for micro benchmarks. The decorator form, where the benchmark fixture is applied to a nested function definition, adds an unnecessary function call, and the README offers the direct call as the better way to get more accurate results for fast code.

def test_my_stuff(benchmark):
    benchmark(time.sleep, 0.000001)  # way more accurate results!

That is a small thing, but it is the kind of detail that decides whether two of your numbers are comparable.

pedantic mode hands you the round and iteration controls

When you need exact control over how the benchmark runs, there is a special mode called pedantic. The README describes it as the way to do fine control over a setup function, exact control of iterations and rounds, and the example passes all of them at once.

def test_with_setup(benchmark):
    benchmark.pedantic(something, setup=my_special_setup, args=(1, 2, 3), kwargs={'foo': 'bar'}, iterations=10, rounds=100)

The parameters are the ones that matter in practice. setup is a function called before each iteration, which is how you get a fresh input state without paying for the setup inside the measured region. args and kwargs are the arguments passed to the function under test. iterations is how many times the function is called in a round, and rounds is how many rounds are run.

Fixing both numbers is the point of pedantic mode. The automatic calibration picks values it thinks are appropriate for the timer and the function, and if you want your result to be reproducible on another machine, or comparable with a number you recorded earlier, pinning iterations and rounds is how you do that.

The pedantic documentation page covers the mode in more detail, and the FAQ page exists for the cases where the automatic behaviour surprises you.

Comparison modes are where the plugin earns its keep

A single timing is rarely the interesting output. The plugin's screenshots section shows the three output modes the README documents: a normal run with the pytest summary, compare mode via --benchmark-compare, and a histogram via --benchmark-histogram, with a note that it also has nice tooltips.

Compare mode is the one that changes how you use the tool. Instead of looking at one run's table, you run a benchmark, save it, and later run it again and ask the plugin to compare the two, so the question becomes whether the change you made moved the distribution rather than whether one number looks different.

There is a second comparison form, pytest-benchmark compare --between, which compares two saved runs directly, and that is the one you want when you have a baseline from another machine or another branch.

The two flags that are named are the ones worth remembering, because they are the difference between a benchmark you can act on and a benchmark you can only look at.

The summary table itself is the other half of the value, since it appears in the pytest output you already read, rather than in a separate report.

Python 3.10 and pytest 8.1 are the floor, and the release cadence is slow

The project metadata is precise about the support window. The version is 5.3.0, released on 2026-08-24, and requires-python is greater than or equal to 3.10. The classifiers list Python 3.10 through 3.14, and both CPython and PyPy, with the IronPython, Jython and Stackless entries present but commented out with a note to uncomment them if you test on those interpreters.

The runtime dependencies are two: py-cpuinfo2 at version 10.1 or above, and pytest at 8.1 or above. That is a small surface, and the pytest floor is worth noticing, because it means the plugin follows pytest's own major version rather than supporting a wide range of them.

The installation is one line.

pip install pytest-benchmark

The documentation is published on readthedocs, with a stable URL for the latest release and a separate one for the master branch that may include documentation fixes, and the project has a Gitter chat badge. Development is driven with tox.

tox

The last push was on 2026-08-23, the day before the 5.3.0 tag, and the repository carries a CHANGELOG.rst, an AUTHORS.rst, a CONTRIBUTING.rst, a.pre-commit-config.yaml and a.bumpversion.cfg, which is the standard layout for a plugin maintained by one person with a release process.

Editorial conclusion

pytest-benchmark fits a team that already runs pytest and wants a performance number attached to an existing test rather than a second harness to maintain. It is the wrong tool for measuring a block of code or a whole program, since the plugin benchmarks functions only, and for very fast code the README itself steers you away from the decorator pattern. Verify first that the timer you calibrate against is one you trust on your machine, because the rounds are sized from that measurement, and check the compare output rather than a single number when you are judging whether a change mattered.

Frequently asked questions

Official sources

1. ionelmc/pytest-benchmark on GitHub
2. Issues
3. License: BSD-2-Clause
4. README
5. Releases