BenchmarkTools.jl: the statistics behind Julia's performance tests

The @benchmark macro and what a Trial actually contains

BenchmarkTools makes performance tracking of Julia code easy by supplying a framework for writing and running groups of benchmarks and for comparing benchmark results. The primary macro is @benchmark, and the result it returns is a Trial, which the documentation describes as a number of samples with a number of evaluations and a full statistical summary rather than a single elapsed time.

The sample output in the README shows what you get: a range from minimum to maximum, the garbage collector share alongside each timing, the median, the mean with a standard deviation, a histogram of the samples, and a memory estimate with an allocation count. The memory and allocation lines matter as much as the time. A function that takes the same time but allocates 1 KiB per call is a different engineering problem from one that allocates nothing, and a framework that reports both in one place is making that comparison easy.

The setup expression is the other thing to understand. It runs once per sample and is not included in the timing results, and the comment in the README notes that each sample can require multiple kernel evaluations. That is the mechanism that lets the framework decide how many repetitions to run to reach a stable estimate, and it is why the number of samples and the number of evaluations are reported separately: the kernel is run many times inside a sample, and the sample is the unit of statistics.

The @benchmark macro is the careful one. For a quick sanity check there is @btime, a convenience wrapper around @benchmark whose output resembles Julia's built-in @time macro, and it accepts a seconds expression that sets a rough time budget.

Interpolation is what keeps a benchmark honest

If your expression depends on external variables, the README says you should use $ to interpolate them, in order to avoid the problems of benchmarking with globals. The mechanism is that any interpolated variable or expression is pre-computed before benchmarking begins and then passed to the benchmark as a function argument.

The three-line comparison in the documentation makes the difference visible without needing to interpret timings. A matrix is bound to a global, then benchmarked three ways: interpolated as $A, interpolated as a fresh $(rand(3,3)), and included directly as rand(3,3) with no interpolation. The README's own comments mark the difference, the first two interpolating the work and the third including the random matrix construction in the measured time.

The point is the first one. Passing a global into a measured expression means the compiler has to look it up at runtime and the measurement includes type instability cost that your real code may or may not have. Interpolating hands the value in as a function argument, which is what you want when you are asking how fast the operation is rather than how fast the global access is.

The second problem is the one the documentation calls out explicitly, and it is subtler. Sometimes inline values in simple expressions give the compiler more information than you intended, causing it to cheat the benchmark by hoisting the calculation out of the benchmark code. The example is @btime 1 + 2, which reports 0.024 nanoseconds, and the README gives the rule of thumb: if a benchmark reports that it took less than a nanosecond to perform, this hoisting probably occurred. The fix is to interpolate the values instead, which turns the same expression into a measurement of an actual runtime addition of two runtime values rather than a constant fold.

Why the package exists: three failure modes in the old design

The history section is the most instructive part of the documentation, because it explains the design by describing what it replaced.

The story begins with two packages, Benchmarks and BenchmarkTrackers. Benchmarks implemented an execution strategy for collecting and summarizing individual benchmark results, while BenchmarkTrackers implemented a framework for organizing, running and determining regressions of groups of benchmarks. Under the hood, BenchmarkTrackers relied on Benchmarks for actual benchmark execution. That combination was used for automated performance testing of Julia's Base library for a while.

Then three problems showed up, and they are worth reading as a specification of what a benchmark framework has to get right.

First, individual sample noise could significantly change the execution strategy used to collect further samples. In other words, the decision about how much to measure was being made from noisy data, which makes the final result depend on luck rather than on the code. Second, the estimates used to characterise benchmark results and to detect regressions were statistically vulnerable to noise, in the words of the documentation. Third, different benchmarks have different noise tolerances, and the old system treated them alike.

That third point is the one that shaped the design most visibly. A microbenchmark of a small allocation-free function and a benchmark of a large allocation-heavy routine do not have the same tolerance for jitter, and a single global policy for both is wrong. The per-benchmark tuning that follows from taking that point seriously is why the manual covers many other features, both for additional output and for more fine-grained control over the benchmarking process.

The old design is also the reason the package is not a thin wrapper. Replacing an execution strategy that was itself unstable required owning the statistics, not just the timers.

The package sits next to two other pieces of Julia's performance infrastructure

The README is careful about scope, and it points at two adjacent projects so that nobody confuses what this package does with the whole of Julia's performance testing.

First, BenchmarkTools is what writes and runs the benchmarks found in BaseBenchmarks.jl, which is the repository of the actual benchmark suite for the language's standard library. So BenchmarkTools is the framework and BaseBenchmarks.jl is a large real-world consumer of it. The documentation points at BaseBenchmarks.jl as the extensive example of a benchmark suite in the real world, and separately mentions a toy benchmark suite in this repository at benchmark/benchmarks.jl, which is what to read first if you just want to see the shape of a suite.

Second, and more important for expectations, the CI infrastructure for automated performance testing of the Julia language is not in this package. It lives in Nanosoldier.jl. That distinction matters if your goal is regression testing across commits, because the package that decides when to fail a pull request on a performance regression is a different repository from the one that measures a benchmark.

The repository layout reflects the same division. The top level holds Project.toml, the source, the test directory, the docs, the benchmark directory, and BenchmarkPlots/, which is a separate directory of plotting helpers rather than part of the core. There is also CITATION.bib, a .JuliaFormatter.toml, a .codecov.yml, and a LICENSE.md whose terms GitHub cannot classify as a standard identifier.

The documentation adds one more practical resource: a series of tips and tricks for benchmarking on Linux, written to help eliminate noise during performance tests. On a shared or virtualised machine, noise is the thing standing between you and a number you can act on, so that page is worth more than it looks.

Installing it and running your first benchmark

BenchmarkTools is a Julia package, and installation happens in the REPL. The documentation says to open Julia's interactive session, press the ] key to enter package mode, and then add the package:

pkg> add BenchmarkTools

That is the whole installation. There is no separate build step and nothing to compile, which is what you would expect from a package that lives in the same registry as the language itself.

The first benchmark uses the macro. The README's quick start looks like this:

julia> using BenchmarkTools

# The `setup` expression is run once per sample, and is not included in the
# timing results. Note that each sample can require multiple evaluations
# benchmark kernel evaluations. See the BenchmarkTools manual for details.
julia> @benchmark sort(data) setup=(data=rand(10))

The `julia>` prefixes are the REPL prompt, so in a script you write the bare expression with the macro and the setup keyword argument, and the result is a Trial you can inspect or compare. The setup keyword holds the data generation outside the measurement, which is the whole point of it: generating ten random numbers in every evaluation would tell you about rand, not about sort.

For a quick check rather than a measurement you would keep, @btime is the wrapper, and it takes a seconds expression for a rough time budget:

# The `seconds` expression helps set a rough time budget, see Manual for more explanation
julia> @btime sin(x) setup=(x=rand()) seconds=3
  4.361 ns (0 allocations: 0 bytes)
0.49587200950472454

Note the two-part output: the timing line first, then the value of the expression. The seconds budget is the knob that decides how long the benchmark keeps running, so it is a cost control rather than a measurement, and setting it well is how you get a stable estimate without waiting indefinitely.

What this package is not, and where the honest limits are

The clearest limit is that BenchmarkTools measures, it does not profile. It answers how long did this expression take, how much garbage did it produce, how many allocations. It does not tell you where a whole program's time goes, and nothing in the documented output suggests it tries. A package that answered both would need different machinery, and the manual's focus on sampling strategy, tuning and comparison reflects that scope.

The second limit is that resistance to noise is not automatic. The package was written to be less sensitive to noise than the system it replaced, not to be immune to it. On a machine with a noisy neighbour, a benchmark that reports a median is still reporting a median of noisy samples. This is why the Linux noise tips exist and why the time budget is worth spending thought on. The statistic protects you from a single wild sample; it does not protect you from a machine that is busy.

The third limit is the one practitioners meet first. If you take a number from a benchmark and put it in a pull request description, you are making a claim about performance, and the framework's output is a summary of repeated runs on one machine at one moment, not a guarantee. The difference between the median, the mean and the range in the Trial output is exactly the spread you need to consider before you claim something.

There is also a scope boundary that is easy to miss. The framework organises and runs groups of benchmarks and compares results, and the automation that turns those comparisons into a decision about whether a commit regressed lives in a different package. If you are building a performance regression gate for a large project, you are assembling BenchmarkTools with something else, and the README is honest that the other part is not here.

Editorial conclusion

Adopt BenchmarkTools.jl whenever you need a number you can defend, because the package exists to make timing results less sensitive to noise and its Trial output gives you a median, a spread, the garbage collector share and an allocation count rather than a single figure. Do not adopt it as a quick profiling substitute, since it measures a specific expression repeatedly rather than telling you where a program spends its time, and do not trust a sub-nanosecond result without checking for hoisting. Before you commit, run the package manager add command, use interpolation with $ for anything depending on a global, and treat the seconds time budget as a cost knob rather than a measurement, then compare against the real-world example in BaseBenchmarks.jl rather than the toy suite in the repository.

Frequently asked questions

Official sources

1. Issues
2. JuliaCI/BenchmarkTools.jl on GitHub
3. README
4. Releases