benchee measures a function three ways, and the memory run is a separate execution

The unit of work is a zero argument function

Benchee's entire interface is a map from a string name to a zero argument function. There is no DSL, no macro to wrap, no scenario object. That is why the README calls it extensible: anything that can be expressed as a function returning a value can be compared against anything else that can.

The example the README gives compares two ways of producing the same flat list:

list = Enum.to_list(1..10_000)
map_fun = fn i -> [i, i * i] end

Benchee.run(
  %{
    "flat_map" => fn -> Enum.flat_map(list, map_fun) end,
    "map.flatten" => fn -> list |> Enum.map(map_fun) |> List.flatten() end
  },
  time: 10,
  memory_time: 2
)

What you should see is a table of statistics and a comparison, not a raw list of timings. The names you choose are the labels in that table and in the HTML report a plugin generates, so naming them well is the only configuration most users need.

Two options do the visible work here. time is the total number of seconds each function is measured for, and memory_time is the duration of the memory pass. Because both are separate passes, the run takes longer than the sum of the obvious parts, and the README's own sample prints an estimated total run time before starting so you know what you have committed to.

Warmup, then a timed loop, then two more measurements

The execution model has three phases and they are not the same measurement.

The first is warmup. The functions run for the warmup duration with results discarded, which the README describes as simulating a warm, running system. The sample configuration prints warmup: 2 s as the default. This phase is the one users most often get wrong: if a function needs more than the warmup to reach steady state, JIT compilation and lazily loaded code land inside the measured window and the reported time is inflated.

The second is the timing pass, which runs each function repeatedly for the time budget and records per iteration run time. Precision is claimed up to nanoseconds and is operating system dependent.

The third covers the other two metrics. Memory is measured in its own shorter pass, sized by memory_time rather than time, which is why the example sets memory_time: 2 while time: 10. Reductions are the third metric, an Erlang virtual machine concept counted separately again with its own reduction_time, which the sample output shows as 0 ns by default. Running memory and reductions in their own passes is what keeps a memory allocation from distorting the timing numbers, and the trade-off is total wall clock time.

The README's feature list also mentions optionally measuring the overhead of function calls, so the reported number is your code's execution time rather than the harness's dispatch cost. That is a correction to the measurement rather than a separate metric.

Statistics that answer the question an average cannot

The default columns are average, ips, deviation, median and 99th %. Each answers something different.

ips is iterations per second and exists only for run time; the README notes it is good for graphing because higher is better. average is the mean. deviation is the standard deviation expressed as a percentage of the average, with raw absolute values available. median is the middle value once sorted, which the README describes as more stable than the average and more likely to be typical. 99th % is the 99th percentile, framed as roughly the worst case since 99 percent of measured values fall below it.

An extended set is available on request and adds minimum and other percentiles. The README prints the available core count and a parallel setting in the run header. It says parallel jobs can gather more results in the same time or simulate a system under load.

Outlier removal is available as an option, and the sample output shows the Excluding outliers line set to false by default. Turning it on changes the reported distribution rather than cleaning it, so a comparison with and without outliers is two different experiments.

Inputs, hooks and pre_check, and what the samples directory reveals

Three extension points carry most of the real use.

Inputs let a benchmark run the same functions over different parameter sets, which is how you find the point where one implementation overtakes another rather than at one arbitrary size. The sample output prints inputs: none specified when none are given.

Hooks run code before and after a benchmarking invocation without that code counting toward the measured time, and they come in scopes: suite hooks, scenario hooks with before_scenario and after_scenario, and function hooks with before_each and after_each. Configuration can be global or local. This is the mechanism for seeding a database, opening a connection or populating a cache before the loop starts, and the README has a complete worked example of the ordering.

Parallel runs are the third point: jobs execute in parallel either to gather more samples in the same wall clock time or to simulate load.

The samples directory documents the surface better than the feature list does. There is pre_check.exs for measuring the cost of the check itself, memory_breaker.exs and memory_changing.exs for showing how memory behaviour changes across runs, repeat_n.exs for fixed repetition counts rather than time, macro_benchmark.exs for the macro based interface, run_parallel.exs, outlier_removal.exs, save_and_load.exs and save_and_report.exs for persistence, and formatters.exs. run_samples.sh at the repository root runs them, which is also the fastest way to see what each option does.

Formatters, plugins and the Erlang entry point

Results are not locked to the console. Formatters turn a run into HTML, markdown, JSON and other formats, and the plugin architecture is what makes that possible: the benchmark engine emits data, a formatter decides presentation. benchee_html is the named example, and it produces an HTML report where individual graphs can be exported as PNG images.

The console formatter itself is configurable, so a CI job does not have to parse the human readable table. Saving, loading and comparing previous runs is a documented feature, which is how you turn a one-off measurement into a regression check: run the benchmark, save the result, load it later and compare. For a library that computes statistics rather than a single number, that is the feature that changes a workflow.

There is also a documented path for calling Benchee from Erlang rather than Elixir, which matters for teams with Erlang code calling into an Elixir codebase.

One structural note from the repository: an architecture/ directory sits at the top level alongside lib/, and the tooling files include.credo.exs and.exguard.exs.

A concrete reading of the sample output, and one oddity in it

The README shows the output of the flat_map example on Linux with an AMD Ryzen 9 5900X, 24 available cores, Elixir 1.19.0 and Erlang 28.1, JIT enabled. Read that header before any number: a benchmark is only comparable to another run on comparable hardware, and the header is what tells you whether two runs are comparable.

In that output flat_map reports 3.96 K ips at an average of 252.74 μs with a deviation of plus or minus 15.64%, and map.flatten reports 1.84 K ips at 543.57 μs with a deviation of plus or minus 44.18%. The comparison line states map.flatten is 2.15x slower with a difference of 290.83 μs. The deviation column is the interesting part: 15.64% against 44.18% is a statement about the machine's noise floor as much as about the two functions, and a 2.15x gap with that much spread is a claim worth re-running rather than shipping.

The memory table shows 624.97 KB for flat_map and 781.25 KB for map.flatten, described as 1.25x memory usage plus 156.28 KB. Immediately below it the sample output prints a line reading All measurements for memory usage were the same. Those two statements do not agree, and the README does not explain when that line appears or what it means for the numbers above it. Anyone relying on memory output should check this in their own runs before drawing a conclusion from that section.

Known issues, maintenance and what a benchmark cannot tell you

The README carries a Known Issues section in its table of contents, which is where a library like this puts the honest limits of what it measures. It does not summarise them in the prose, so the reader has to open the documentation for that.

The structural limitation is more useful to state plainly. A benchmark that touches the network, a database or a file on disk is measuring that dependency, and the harness has no way to subtract it. The README's features never claim otherwise; there is no mention of distributed runs, of coordinating multiple machines, or of containerised isolation.

Maintenance facts are healthy. Releases 1.4.0 on 2025-04-14, 1.5.0 on 2025-10-21 and 1.5.1 on 2026-06-06 show a roughly six month cadence on a 1.x line, the repository is not archived, and the last push was on 2026-10-05. CHANGELOG.md at the root and the Hex documentation linked in the badges are the places to look before pinning a version.

The licence is MIT, with LICENSE.md at the root. No legal advice is offered. For a test-only dependency in a mix.exs, that is the least interesting part of the decision, which is appropriate.

Editorial conclusion

Use benchee when the question is which of two or more Elixir functions is faster, how much faster, and how much that varies between runs, since the median, deviation and 99th percentile columns answer questions an average alone cannot. Verify first that your machine is quiet during the run, and raise the warmup above the default two seconds before trusting a number, since the first iterations are what the warmup exists to discard.

Frequently asked questions

Official sources

1. bencheeorg/benchee on GitHub
2. Issues
3. License: MIT
4. README
5. Releases