clpeak and running one benchmark across every device API

What clpeak measures, and the point of running it everywhere

clpeak is a compute latency peak tool. It measures the peak compute throughput, memory bandwidth and latency that CPUs, GPUs and NPUs actually reach, using small, tight kernels alongside each vendor's own tuned libraries. It reaches a device through every API that exposes it and runs the same tests on each, so the numbers from one machine can be compared side by side.

That last clause is the design. A single device with an XDR display, an M-series GPU and a Neural Engine exposes all three through different APIs, and each vendor's own benchmark reports a different figure in a different unit, so they cannot be compared at all. clpeak runs the same test through every route and puts the answers in one table.

The backend list is the size of the project. For GPU compute there is CUDA, ROCm and HIP, Metal, oneAPI and SYCL, Vulkan and OpenCL. For AI runtimes across NPU, GPU and CPU there is ONNX Runtime, Core ML and LiteRT. And for the CPU there are native kernels picked per instruction set at run time, for x86-64 and AArch64.

The engine ships as a command-line tool and as an app for Windows, macOS, Linux and Android, which is a larger platform claim than most tools of this kind make, and the badges confirm the Google Play and Snap Store distribution.

The primary language is C++, the licence is GPLv3, the homepage is krrishnarraj.github.io/clpeak, and the recent releases are 3.0.2-b1 from 2026-10-15, 3.0.1 from 2026-10-04 and 3.0.0 from 2026-09-26, so the 3.0 line is new and the project is moving quickly. The last push was on 2026-10-20.

The audience is a hardware or runtime evaluator, and the screenshot in the readme makes the intent concrete: an Apple M1 Pro with eleven devices across seven backends, the Neural Engine selected, showing its matmul and convolution rates per weight format.

What the output looks like, and why the units differ

The sample output is the most instructive part of the documentation, because it shows how differently the same concept is reported by different stacks.

On an NVIDIA GeForce RTX 5060 through CUDA, the tensor core numbers are given per weight format: fp16 at 42.6 TFLOPS, fp8_e4m3 at 85.2 TFLOPS, nvf4_e2m1 at 328 TFLOPS and int8 at 84.7 TOPS. Then the cuBLASLt GEMM peak, which is the tuned library rather than the raw instruction path, gives fp16 at 79.1 TFLOPS and nvf4_e2m1 at 300 TFLOPS. Then global memory bandwidth at 420 GB/s for float4, and kernel launch latency as a roundtrip of 6.35 microseconds.

On the Apple Neural Engine through Core ML, the same categories come out in different forms: Core ML matmul peak with fp16 at 10.4 TFLOPS, int4_lut at 11.3 TFLOPS and int8_qdq at 11.5 TOPS, then a transformer block prefill at 4.99 TFLOPS and decode at 43.2 GB/s, and finally Core ML dispatch latency of 781 microseconds for a trivial operation.

Two things fall out of that comparison. The first is that the low-precision formats are where the dramatic numbers live, with a factor of nearly eight between fp16 and nvf4_e2m1 on the tensor cores, and the second is that the Neural Engine's numbers are in a different unit for the same idea, TFLOPS for the matmul peak and GB/s for the decode bandwidth, which is the transformer decode path rather than a raw throughput.

The dispatch latency figure of 781 microseconds against a 6.35 microsecond kernel launch roundtrip is also the most useful single number in the readme for anyone building a real-time application, and it is the kind of figure that only exists because the same test runs everywhere. A benchmark that only measured one API could not surface the gap.

The sample output is condensed from the saved runs in the results/ directory, which the readme later describes as the baselines to check a suspicious number against.

Building it, and which backends appear depends on your SDKs

The build is a standard CMake flow with a recursive clone:

git clone --recursive https://github.com/krrishnarraj/clpeak
cd clpeak
cmake -S . -B build
cmake --build build -j
./build/clpeak

The --recursive flag is there because the repository has a .gitmodules entry, and the third_party/ directory in the tree suggests the dependencies are vendored that way.

The conditional compilation is the interesting part. A backend is built when its SDK is found, and -DCLPEAK_ENABLE_<BACKEND>=OFF leaves it out. So the binary you get is a function of what is installed on your machine, and the list of backends in a given build tells you what that machine could reach.

Two backends break that pattern. The ONNX Runtime and LiteRT backends need no SDK, because their runtimes are loaded when clpeak runs, and --onnx-lib and --litert-lib choose which library to load. That is a better design than a build-time dependency for runtimes that are distributed as single libraries, and it means the same binary can pick up whichever runtime version you point it at.

The desktop app is built alongside the CLI when the Flutter SDK is on PATH, into build/clpeak-gui/. The project page lists every option, and the Android and iOS builds are documented in app/AGENTS.md.

The project is unusually explicit about being written for coding agents. The tree is documented with AGENTS.md files, the root one covering the architecture, the directory map and the conventions for adding a benchmark or a backend, with a subdirectory's own AGENTS.md covering the code in it. NOTES.md lists the workarounds for runtime bugs that are waiting on an upstream fix, which is the sort of file that saves an afternoon when a number comes out wrong for a reason that is not your fault.

The command line, and comparing two runs

The usage section is a list of one-line invocations, and reading them together tells you the shape of the tool.

Running clpeak with no arguments does every test on every device. --list-devices shows what clpeak can see, named the way the --devices option takes them, so you can discover the device identifiers without guessing at the naming scheme. --devices takes a comma separated list such as cuda:0,vulkan:1, and the backends can also be selected individually with --vulkan or --opencl.

The test categories are selectable too. --bandwidth runs one category, and the list is --compute, --bandwidth, --latency, --ai, among others. --gemm runs one specific test on every backend that has it, which is the comparison that matters when you are deciding between APIs, and --describe prints what each test and each reading actually measures.

That last flag is the one to know about. Every number in this tool is meaningless without knowing what it measures, and --describe is the built-in answer to that question rather than a documentation page you have to go and find.

Two flags handle persistence and comparison. -o saves the run as JSON, and --compare takes a saved run and compares against it, so clpeak --compare run.clpeak.json runs the benchmark again and diffs it against the earlier result. That is what turns a one-off measurement into a regression check, and it is why the results/ directory in the repository exists with committed runs to compare against.

The tool also has a bug reporting path that produces the same artefact. For the CLI, run clpeak --verbose -o report.clpeak.json and attach that file to an issue, or, if clpeak crashed before finishing, attach report.clpeak.log from the same folder, which is written as the run goes. The app has the same idea behind Settings, Diagnostics and Verbose diagnostics, with an Export JSON on the results page, and a run the app did not survive is listed under History as Runs that did not finish with its own export.

Running the release binaries, and what the peaks do not tell you

Each release has a zip per platform with the CLI and the app, plus cuda, rocm and oneapi variants that add those backends, and a dmg of the macOS app. The binaries are not signed with a developer certificate, so the operating system asks before running them the first time, and the readme gives the per-platform procedure.

On Windows, before extracting you right-click the zip, choose Properties, tick Unblock and confirm, and if SmartScreen still complains you choose More info and Run anyway. On macOS you open the dmg and drag clpeak to Applications, and the first launch is refused until you go to System Settings, Privacy and Security and click Open Anyway next to clpeak, or you clear the quarantine flag from a terminal:

xattr -dr com.apple.quarantine /Applications/clpeak-gui.app
xattr -dr com.apple.quarantine ~/Downloads/clpeak-*-macos-arm64

On Linux you can install the snap with `sudo snap install clpeak --classic` or use the zip as it is, and on Android it is on Google Play.

The substantive limit is what a peak number means. Every figure in this tool is the best a device can do on a small, tight, vendor-tuned kernel. That is a real and useful number for choosing hardware or deciding which API to build on, and it is not a prediction of how your application will perform, because real workloads are larger, less regular and usually not shaped like a GEMM. A tool that reported only peaks and you read as if it reported throughput would mislead you.

The other limit is that the comparison is only as fair as the tuning behind each number. The CUDA path reports both the raw tensor core instruction path and the cuBLASLt library path, and they differ by a factor of nearly two for fp16, which is a fair illustration that the number you get depends on how the work is expressed.

The alternative to compare against is simply running your own workload, which is the only measurement that predicts your application. clpeak's value is that it tells you the ceiling and the relative cost of an API, and the --compare flag makes it usable as a hardware change detector rather than a one-off curiosity.

Editorial conclusion

clpeak is a good fit for someone choosing hardware or a runtime, because the same test is run through every API a device exposes, so the CUDA number and the Core ML number on one machine are directly comparable, and the saved-run JSON gives you a way to re-measure and diff. It is a poor fit if you want to know how fast your application will run, since every number here is a peak measured with tuned vendor libraries rather than your own workload. Before drawing conclusions, read the --describe output to see what each reading measures, and check the results/ baselines that ship with the project, since the readme points at them as the reference for a suspicious number.

Frequently asked questions

Official sources

1. krrishnarraj/clpeak on GitHub
2. License: GPL-3.0
3. Project website
4. README
5. Releases