JMH: the OpenJDK harness that makes Java benchmarks mean something

Why hand-rolled Java timing loops are wrong

JMH is described as a Java harness for building, running and analysing nano, micro, milli and macro benchmarks written in Java and other languages targeting the JVM. That phrase is doing more work than it appears to. The harness exists because a timing loop you write yourself measures the wrong thing, usually by measuring dead code elimination, an incomplete JIT warmup or a constant-folded result rather than your code.

The fix is that JMH generates and runs the benchmark code itself, and it will not let you take the easy version. It forks a fresh JVM per iteration by default, warms up before measuring, and keeps the result alive so the compiler cannot remove the work. None of that is exotic; it is just discipline applied mechanically, which is why the README's framing is that JMH makes avoiding benchmarking pitfalls easier rather than impossible.

The last point in the README is the one worth remembering. It says plainly that a nice harness will not free you from considering benchmarking pitfalls, and that benchmarks should be peer-reviewed. A benchmark that measures something nobody will run is still wrong, and the harness cannot tell you that.

Four code generators behind one annotation

The repository tree explains the architecture better than the README does, because the module names line up with the mechanism. There is jmh-core for the harness itself, and then four generator modules: jmh-generator-annprocess for annotation processing, jmh-generator-asm for generating bytecode directly, jmh-generator-bytecode for manipulating existing bytecode, and jmh-generator-reflection for the reflection-based path. Alongside them sit jmh-samples, jmh-archetypes, jmh-ant-sample, jmh-core-benchmarks for JMH's own benchmarks, jmh-core-ct for compile-testing and jmh-core-it for integration tests.

That four-generator split exists because a benchmark is not just a method call. It has to be wrapped in generated infrastructure that creates the forked JVM, drives the warmup and measurement iterations, and reports results in the format the reporting layer understands. Doing that with plain Java source and an annotation processor is the portable route, which is why the samples and the archetypes are built around annotations. The other generators exist for cases where the annotation processor cannot reach, which is why having all four is a maintenance cost the project accepts.

Because the generated code is the benchmark, the build has to run the generator. That single fact drives the rest of the README, including its insistence that you cannot simply drop the jmh-core jar into a build and start running.

The archetype workflow, step by step

The README recommends Maven with a standalone project that depends on the jar files of your application, and calls this preferred because it keeps benchmarks correctly initialized and produces reliable results. Running inside an existing project or an IDE is possible but described as more complex and less reliable. Generation is a single archetype command:

$ mvn archetype:generate \ -DinteractiveMode=false \ -DarchetypeGroupId=org.openjdk.jmh \ -DarchetypeArtifactId=jmh-java-benchmark-archetype \ -DgroupId=org.sample \ -DartifactId=test \ -Dversion=1.0

Step two builds it, and step three runs it:

$ cd test/ $ mvn clean verify

$ java -jar target/benchmarks.jar

The self-contained jar holding the benchmark and all JMH infrastructure is the deliverable, and running it with -h lists the command line options. Note the README's warning that the current and target folder should not already contain a Maven project, or generation will not work reliably.

Benchmarking another JVM language means picking a different archetype artifact id, which usually amounts to replacing java in the name, and possibly adjusting the generated pom.xml. The archetype catalog at repo.maven.apache.org has the full list. For a large codebase the README's advice is to keep benchmarks in a separate sub-project that depends on the tested modules through ordinary build dependencies, which keeps the generator configuration from leaking into your production build.

IDEs are supported and still discouraged

The IDE section exists because people ask, and it walks through the same three steps with a warning attached. Running benchmarks from an IDE is generally not recommended, the README says, because of the generally uncontrolled environment in which the benchmarks then run.

The mechanics still work. Most IDEs create the Maven project from the archetype through a GUI, and you need to make sure the IDE knows about the Central archetype catalog and can find `org.openjdk.jmh:jmh-${lang}-benchmark-archetype`. Building is mostly automatic from the Maven configuration, with IDEA and NetBeans needing little or no effort, while Eclipse may need annotation processors enabled explicitly. Running is the awkward part: there is no direct support, so you write a main method that invokes the JMH Java API, and the samples show how.

The most emphatic warning in the whole README sits in this section. JMH is not intended to be used the way a testing library such as JUnit is, and simply adding the jmh-core jar to a build is not enough to run benchmarks. If you take one thing from the README, take that: the harness is not a dependency you add, it is a build configuration you generate.

Where the build stops and community bindings begin

JMH ships Maven build scripts and nothing else, which is stated directly: the project does not ship build scripts for build systems other than Maven. Two community-supported bindings fill the gap, a Gradle JMH Plugin and a Scala SBT JMH Plugin. For Ant users there is a jmh-ant-sample module in the tree that describes the steps needed to build a benchmark project by hand.

That division has consequences for a team. Maven users get the archetype path the README treats as the golden configuration. Gradle users get a plugin maintained by somebody else, which means JMH version support depends on the plugin's author rather than on the OpenJDK project. The Ant path is a sample, so you are following documentation rather than using a supported configuration.

The README's advice for diagnosing problems fits this structure. It tells you the archetypes provide the golden build configuration, so generating a clean benchmark project and transplanting your benchmark there is the first thing to try, particularly when upgrading to newer JMH versions, since small differences in build configuration are often the actual cause of a failure. It also advises running the latest JMH release or bleeding edge, on the reasonable grounds that current development code is usually leaner and more correct.

Support paths and the licensing situation

Bug reports go to the OpenJDK bug system under a specific path: project CODETOOLS, component tools, sub-component jmh. The alternative is the jmh-dev mailing list, and the README asks you to check the mailing list archives before reporting, since the question may already be answered.

Before any of that, the README lists prerequisites that read more like a benchmark methodology checklist than support instructions. It points you at the JMH annotations Javadocs and samples as essential reading, and at the samples specifically for API use cases, culprits and pitfalls. It asks you to make sure your benchmarks are peer-reviewed. It also suggests checking whether the issue is already discussed. Only after all of that is it appropriate to file a report.

On licensing, the repository is GPL-2.0, which is the standard OpenJDK licence, and the tree carries both a LICENSE file and a THIRD-PARTY file recording third-party components. There are no tagged releases in the repository, so version tracking for JMH happens through Maven coordinates and the release announcements rather than a GitHub releases list. The repository is not archived and the last push was on 2026-07-14, with five open issues, which for a project of this age and centrality suggests a stable maintenance state rather than an abandoned one.

Editorial conclusion

JMH is one of the few pieces of JDK tooling that is genuinely hard to use incorrectly once set up properly, and the setup is the hard part. The archetypes generate a build with the annotation processor wired correctly, which is why the README treats a hand-rolled Maven block as a mistake rather than a shortcut. For large projects the useful pattern is a separate sub-project that depends on the modules under test, which keeps the benchmark build configuration out of your application build. Read jmh-samples first, since the comments there carry run instructions the README points you to repeatedly, and keep the harness for JVM-level questions rather than end-to-end timing. The repository is not archived, the last push was on 2026-07-14, and bug reports go to the OpenJDK CODETOOLS project or the jmh-dev mailing list.

Frequently asked questions

Official sources

1. Issues
2. License: GPL-2.0
3. openjdk/jmh on GitHub
4. README