sbt-jmh: running JMH microbenchmarks from an sbt project without the generated code

What the plugin actually saves you

sbt-jmh is an sbt plugin for running OpenJDK JMH benchmarks, and its tagline is trust no one, bench everything.

JMH describes itself as a Java harness for building, running and analysing nano, micro, milli and macro benchmarks written in Java and other languages targeting the JVM. The reason a plugin exists at all is the note in the documentation that JMH has a very specific way of working, since it generates loads of code. A hand written JMH benchmark is a class with a state field, a benchmark method and a set of annotations, and it is compiled by a separate build that generates the harness around it. That separate build is the friction.

The plugin removes it. You write your benchmarks in src/main/scala, they are picked up and instrumented by the plugin, and you run them with Jmh/run. The JMH generator still happens, but you never see the generated code and you never maintain a second build definition.

The plugin is Apache-2.0 licensed, the primary language is Scala, and the last push to the repository was on 2026-10-19, with v0.4.8 published in September 2025.

Enabling the plugin in a multi-project build

Since sbt-jmh is an AutoPlugin, activating it means adding one line to your project/plugins.sbt file:

// project/plugins.sbt
addSbtPlugin("pl.project13.scala" % "sbt-jmh" % "0.4.8")

And enabling it in the projects where you want it, which the documentation notes is useful in multi-project builds since you can enable it only where you need it:

// build.sbt
enablePlugins(JmhPlugin)

If you define your project in a Build.scala rather than a build.sbt, you also need the import:

import pl.project13.scala.sbt.JmhPlugin

That per-project enablement is the detail worth noticing. In a large multi-module build, turning the plugin on everywhere means every module gets benchmark infrastructure it does not need, and the plugin's own settings are scoped to the projects that enable it, which keeps the blast radius small.

The plugin's own repository layout reflects that it is a well structured sbt plugin, with a plugin directory holding the implementation, a project directory, a tester project named sbt-jmh-tester for the scripted tests, a notes directory, a build.sbt, a version.sbt, an sbt-jmh.properties file and a RELEASING.md, plus a .java-version file pinning the JDK the plugin builds against.

The version table is the compatibility contract

The README carries a version table that pairs each plugin release with the JMH version it defaults to, and it is the most useful thing in the documentation because it tells you what you get without configuring anything.

Plugin 0.4.8 works with sbt 1.3.0 or 2.0 and defaults to JMH 1.37, as do 0.4.7 and 0.4.6. The 0.4.5 and 0.4.4 lines default to JMH 1.36. Going back, 0.4.3 defaults to 1.32, 0.4.2 to 1.31 with a note that the async profiler supports 2.x, 0.4.1 to 1.30, and 0.4.0 to 1.25 with the note that the profilers moved into the JMH core.

The older entries show how the plugin tracked sbt itself over the years, with 0.3.x releases supporting sbt 0.13.16 and 0.13.17 alongside sbt 1.0 and 1.1.x. The 0.3.7 and 0.3.6 lines are marked as supporting JDK 11, 0.3.4 as adding GraalVM support, and 0.3.0 as the release that brought the async profiler and flame graphs.

The guidance is to always use the newest version that has the JMH version you need, with the note that uninteresting versions are skipped in the listing, and that you should stick to the latest version at all times anyway.

Running a benchmark and the iteration discipline it insists on

Once the plugin is enabled you type Jmh/run, and all the JMH options work as expected, with Jmh/run -h printing the help. A worked example in the documentation is:

Jmh/run -i 3 -wi 3 -f1 -t1 .*FalseSharing.*

The explanation is that this means 3 iterations, 3 warmup iterations, 1 fork and 1 thread.

And then the plugin makes a point that the example itself contradicts. Benchmarks should usually be executed at least in 10 iterations as a rule of thumb, but more is better, and for real results the recommendation is to at least warm up 10 to 20 iterations and then measure 10 to 20 iterations again. Forking the JVM is required to avoid falling into specific optimisations, with the parenthetical that no JVM optimisation is really completely predictable.

That is the whole reason this plugin exists, arguably. The common failure mode of JVM benchmarks is a number that is an artefact of the JIT having warmed up the method before the measurement started, or of the compiler having specialised the call site for the exact arguments used. The measurement flags in the example are the knobs that control those effects, and the documentation is telling you the defaults in the example are there to demonstrate the syntax, not to be used.

The plugin also links the nanotrusting nanotime article and other blog posts on microbenchmarking, with the instruction to make sure your benchmark is valid before you set out to implement it, which is a reasonable way to frame a plugin that is mostly plumbing.

Benchmarks in a test source directory

The last configuration topic in the README is the awkward one, and it is worth reading before you set up a benchmark module in a multi-module build.

The plugin expects benchmarks in src/main/scala. If your benchmark should be a module in a multimodule project and needs access to another module's test classes, you might want to define your benchmarks in src/test as well, and the reason given is that IntelliJ does not support compile-to-test dependencies, with a link to the issue tracker entry.

This is not directly supported, the documentation says, but it can be achieved with some tweaks. Assuming the benchmarks live in a module named bench and need access to test classes from anotherModule, the dependency is defined in build.sbt:

lazy val bench = project
  .dependsOn(anotherModule % "test->test")
  .enablePlugins(JmhPlugin)
  .settings(
     Jmh / sourceDirectory := (Test / sourceDirectory).value,
     Jmh / jmhBytecodeDirectory := (Test / classDirectory).value,
     Jmh / dependencyClasspath := (Test / dependencyClasspath).value,

The three settings are the mechanism. The source directory is redirected to the test sources, the bytecode directory the plugin writes the generated harness into is redirected to the test class directory, and the dependency classpath is taken from the test classpath so the benchmark can see the test classes. The comment in the truncated block explains that the benchmarks have to be compiled before the generator scans for them, which is the ordering constraint that makes this fiddly in the first place.

So the plugin is happy with benchmarks in the normal source directory, and the twist is only needed when a benchmark needs to see test-only code.

Where sbt-jmh is the wrong choice

The first limitation is that it is an sbt plugin for one build tool. A Maven or Gradle project that wants JMH uses the JMH Gradle plugin or the JMH Maven plugin, and this has no bearing on it. The plugin is also Scala-first in the sense that it is published for the Scala sbt ecosystem, and while JMH itself supports benchmarks in other languages, the ergonomics of the plugin are for Scala projects.

The second is the version matrix, which is a real constraint rather than a detail. The JMH version is fixed per plugin release, and 0.4.8 ships JMH 1.37. If you need a different JMH version, you are either choosing a different plugin release, which may mean an older sbt or an older JMH, or you are forking the plugin to change the default. There is no single line in a build file that selects a JMH version here.

The third is the one the plugin is most candid about, and it is the reason to read its documentation before using it. The default invocation will give you a number, and that number may be wrong, because the iteration counts and the forking behaviour have to be set deliberately to get a result that survives scrutiny. A team that treats Jmh/run with no arguments as a benchmark result will produce numbers that look precise and are not, and the plugin's own instructions are the only defence against that. The tooling does not make correct microbenchmarks easy, it makes them runnable, and the running is the easy part.

The alternative comparison is with the plain JMH Gradle or Maven plugins, which are more actively maintained and used far more widely, and with ScalaTest or a timing library for the case where you are measuring throughput rather than latency, where JMH's machinery is more than the measurement needs.

Editorial conclusion

sbt-jmh fits a Scala team that already uses sbt and wants microbenchmarks to live in the same build, with the JMH harness generated rather than hand written, and that is willing to follow the iteration and forking discipline the documentation insists on. It does not fit a project that wants a one command benchmark run with default settings, and it is not a way to get a reliable number quickly. Before adopting it, pick the plugin version whose bundled JMH version matches what you need, since the mapping is fixed per release, and treat the advice to warm up and measure ten to twenty iterations with forking enabled as part of the plugin rather than as an optional extra.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. Releases
5. sbt/sbt-jmh on GitHub