Criterion: microbenchmarking Haskell with statistics you can trust

What makes a benchmark number trustworthy

Criterion is described as a library that makes accurate microbenchmarking in Haskell easy, and the readme's subtitle, reliable performance measurement, is the claim in one line.

The way it earns that claim is statistical rather than procedural. The feature list names four things: a simple API that hides a lot of automation and details you should not need to worry about, high-resolution analysis that can accurately measure operations running in as little as a few hundred picoseconds, output to active HTML with JavaScript charts as well as CSV and JSON, and measurements that are cross-validated so that sources of significant noise, usually other activity on the system, can be identified.

The fourth point is the one most benchmarking tools skip. A number produced on a machine that was also running a build is not a measurement of your code, it is a measurement of your code plus whatever else was happening. Criterion measures repeatedly and looks for the outliers and the noise sources, which means a run that was disturbed says so rather than quietly reporting a wrong figure.

The linear regression model is the other feature that earns its place. The readme describes it as allowing measurement of the effects of garbage collection and other factors, and the reports show a regression line fitted through the raw measurements. A function whose timing grows with each iteration is allocating; a function whose timing is flat is not. You can see that difference directly in the report rather than inferring it.

The library was written by Bryan O'Sullivan and is maintained by Ryan Scott, and the readme asks for bugs through the GitHub issue tracker. The homepage is the Hackage page for criterion, the licence is BSD-2-Clause, and the last push to the repository was on 2026-07-02.

A first benchmark, and what the output shows

The tutorial's example is the deliberately ridiculous fib function, benchmarked at four input sizes.

import Criterion.Main

-- The function we're benchmarking.
fib :: Int -> Int
fib m | m < 0     = error "negative!"
      | otherwise = go m
  where
    go 0 = 0
    go 1 = 1
    go n = go (n - 1) + go (n - 2)

-- Our benchmark harness.
main = defaultMain [
  bgroup "fib" [ bench "1"  $ whnf fib 1
               , bench "5"  $ whnf fib 5
               , bench "9"  $ whnf fib 9
               , bench "11" $ whnf fib 11
               ]
  ]

defaultMain takes a list of Benchmark values, each describing one function to benchmark, and it parses the command line so you can run a subset of them. bgroup groups related benchmarks under a common name, bench names one case, and whnf means the benchmark measures to weak head normal form, that is, the value of the result rather than the work of producing it.

The output for each case is the part worth reading carefully:

benchmarking fib/9
time                 1.219 μs   (1.214 μs .. 1.228 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.219 μs   (1.216 μs .. 1.223 μs)
std dev              12.43 ns   (9.907 ns .. 17.29 ns)

The time line gives an estimate of the time for one execution with a confidence interval in parentheses, so 1.219 microseconds with a range of 1.214 to 1.228. The R squared figure is the goodness of fit of the regression, and at 1.000 it means the measurements sit tightly on the line, which is what you want to see before believing the number.

The std dev line is the one that carries the caveat. For fib/9 the standard deviation is 12.43 nanoseconds against a mean of 1.219 microseconds, and the range in the parentheses is wide, from 9.907 to 17.29 nanoseconds. For the earlier fib/1 case the report says variance introduced by outliers: 36%, moderately inflated. Criterion is telling you that the noise in this measurement is a significant fraction of the signal, and a benchmark whose variance is inflated should not be used to judge a 10 percent change.

The two charts and the table underneath them

The report layout is described in the tutorial and is worth understanding, because reading the wrong number is the common mistake.

A report begins with a summary of all the numbers measured, and underneath is a breakdown of every benchmark with two charts and some explanation.

The chart on the left is a kernel density estimate of the time measurements, graphing the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred, and its height indicates how often that measurement was repeated.

The readme explains why a KDE rather than a histogram, which is the more popular alternative. To get good information out of a histogram you have to choose a suitable bin size and width, which the readme calls a fiddly manual task. A KDE is likely to be informative immediately, with no configuration required.

The chart on the right contains the raw measurements from which the kernel density estimate was built. The x axis is the number of loop iterations and the y axis is the measured execution time for that many iterations, and the line behind the values is a linear regression generated from the data. Ideally all measurements will be on or very near that line.

Underneath each chart is a table of the same numbers in text form, with a lower bound, an estimate and an upper bound for each row. The OLS regression row estimates the time needed for a single execution of the activity being benchmarked, and the R squared goodness-of-fit row is the result of the linear regression run on the measurements displayed in the right-hand chart. Then come the mean execution time and the standard deviation rows.

The relationship between the regression estimate and the mean is the interesting part. The regression extrapolates back to zero iterations to estimate a single execution, while the mean averages everything that was measured, including the iterations that carried the warm-up and the allocation churn. For a function that allocates, the two numbers differ noticeably, and the regression figure is the one that answers the question you actually have.

Output is not limited to the terminal. The readme mentions output to active HTML with JavaScript charts, CSV and JSON, and says you can write your own report templates to customise exactly how your results are presented.

The example programs and the shape of the repository

The examples directory is the fastest way to see what kinds of benchmark the library supports, and the file names describe the cases.

Fibber.hs is the tutorial example. Maps.hs is the one to look at if you are benchmarking container operations, since Maps is the standard Haskell map and its performance characteristics are a common subject of optimisation work. Overhead.hs measures the overhead of the benchmark harness itself, which is the check you want when a result looks too good. Comparison.hs compares several implementations against each other. Judy.hs benchmarks against a different data structure. Quotes.hs exercises a different kind of value. ConduitVsPipes.hs compares two streaming libraries. ExtensibleCLI.hs shows how to extend the command line interface. And BadReadFile.hs and GoodReadFile.hs are paired deliberately, the pair being a demonstration of how a benchmark can be made artificially bad or artificially good and how Criterion's statistics reveal the difference.

The repository listing matches a mature Haskell package. There is a criterion.cabal at the root with cabal.project, cabal.haskell-ci and stack.yaml, so the package is buildable with Cabal, with the Haskell CI configuration for the wider ecosystem, and with Stack. Criterion.hs and the Criterion/ directory hold the library, criterion-measurement/ is a separate package for the measurement core, app/ holds an executable, templates/ holds the report templates that the customisable output feature depends on, tests/ holds the test suite, and www/ holds the documentation pages that the Hackage site renders.

There is also a CHANGELOG.md, a LICENSE, a Setup.lhs, and a .gitmodules file, which suggests some part of the build is pulled in as a submodule. The package is published on Hackage as criterion, and the repository publishes no GitHub releases, so the changelog in the repository is the release history.

Where Criterion is the wrong choice

The first limitation is that it measures code in isolation. A Criterion benchmark is a function called in a loop in a controlled process. It tells you how fast that function is when the garbage collector, the cache and the rest of the machine are in a particular state, and it says nothing about how your service behaves under concurrent load with a real request mix. For questions about production behaviour you need a profiler and a load test, and Criterion is not either.

The second is that it is slow on purpose. The statistics come from many repetitions of each case, with outlier detection and cross-validation on top, and the readme warns that the first run of a benchmark program may take a while since the library has to be downloaded and compiled. A Criterion run of a suite that would take a second under a stopwatch can take minutes, which is a real cost in a CI pipeline that runs benchmarks on every pull request.

The third is that the numbers need interpretation to be worth anything, and Criterion does not remove that burden, it just gives you better raw material. The R squared figure has to be read before you trust the estimate, the variance introduced by outliers has to be checked, and the confidence interval has to be compared against the size of the change you are claiming. A benchmark that shows a 5 percent improvement with a 20 percent confidence interval has told you nothing, and that is still true with Criterion in the loop.

The fourth is the tight coupling to Haskell's evaluation model. The whnf and nf variants of the benchmark functions exist because of how lazy evaluation interacts with measuring. For a strict language or for code with different laziness characteristics, the subtleties Criterion handles for Haskell are exactly the ones you would not know to look for. That is a feature for a Haskell user and a non-issue outside the language, but it means the library's design does not travel.

The fifth is that a benchmark suite is only as good as its cases. Criterion can tell you precisely how much faster the new implementation is, and it cannot tell you that the old implementation was the bottleneck. Choosing what to measure remains the hard part, and the library's sophistication makes it easy to spend a lot of time refining a measurement of something that did not matter.

Editorial conclusion

Criterion fits a Haskell developer who needs to tell a real improvement from measurement noise, since the library runs each case many times, cross-validates to spot sources of noise, and reports an estimate with a confidence interval and an R squared figure rather than a single number. It is the wrong choice if you want to profile a whole application or load-test a service, because Criterion measures isolated code in a loop, and it is the wrong choice if you need the benchmark run to be fast, since the statistics come from many repetitions. Before adopting it, look at the examples directory, where Fibber.hs, Maps.hs, Overhead.hs and Comparison.hs show the shapes of benchmark you can write, and note that the last push was on 2026-07-02 with the library published on Hackage under a BSD-2-Clause licence.

Frequently asked questions

Official sources

1. haskell/criterion on GitHub
2. Issues
3. License: BSD-2-Clause
4. Project website
5. README