Cannoli instruments the QEMU JIT so tracing costs almost nothing

Three components, and the split is the whole idea

Cannoli is a high-performance tracing engine for qemu-user, and it can record a trace of both PCs executed and memory operations.

It consists of three pieces, and the division is the design.

The first is a small patch to QEMU that exposes locations to inject some code directly into the JIT. The second is a shared library which is loaded into QEMU to decide what and how to instrument. The third is a final library which consumes the stream produced by QEMU in another process, where the analysis can be done on the trace.

The reason for that split is stated plainly. Cannoli is designed to record this information with minimum interference of QEMU's execution. In practice, that means QEMU needs to produce a stream of events and hand them off very quickly to another process to handle more complex analysis of them. Doing the analysis during the execution of the QEMU JIT itself would dramatically slow down execution.

That is the whole performance story in one paragraph. A tracer that runs its analysis inside the emulator pays the cost of the analysis on every event. A tracer that streams events out pays only the cost of producing them.

The capability claims follow from it. Cannoli can handle billions of target instructions per second, it can handle multi-threaded qemu-user applications, and it allows multiple threads to consume the data from a single QEMU thread to parallelize the processing of traces.

The performance graph in the README shows 2.2 billion instructions per second, and the conditions are stated: a single QEMU thread running the benchmark example on an Intel Xeon Silver 4310 at 2.1 GHz, with the target being mipsel-linux and a hot loop of unrolled nops to benchmark PC tracing bandwidth, described as the worst case for them.

That last detail matters. The benchmark is deliberately the least favourable workload, a tight loop of no-operations that produces nothing but trace events, which is the right thing to publish.

The project is Rust, GPL-2.0 licensed, with the last push on 2026-09-24 and no GitHub releases.

Building it means patching QEMU at a specific commit

The getting started section is unusually concrete about the QEMU side, because the tool is tied to the internals of a particular QEMU version.

First you build Cannoli:

git clone https://github.com/MarginResearch/cannoli
cd cannoli
cargo build --release

Then you check out QEMU:

git clone https://gitlab.com/qemu-project/qemu.git

And then you switch to the current QEMU branch they are working on:

git checkout 78385bc738108a9b5b20e639520dc60425ca2a5a

The commit hash is the important part. A tracing engine that injects code into the JIT is not version-independent, and the documentation pins an exact commit rather than a branch name or a release.

Then the patch is applied:

cd qemu
git am --3way </path/to/cannoli>/qemu_patches.patch

The patch is applied with git am and the 3-way merge option, which is the right tool when the patch was generated against a slightly different tree.

Then you configure QEMU for your targets:

./configure --target-list=mipsel-linux-user,riscv64-linux-user --extra-ldflags="-ldl" --with-cannoli=</absolute/path/to/cannoli>
make -j48

The configure line shows three things at once. The target list, here mipsel-linux-user and riscv64-linux-user, which shows the tool is target-agnostic across architectures rather than tied to one. The extra linker flags adding -ldl, which the shared library loading needs. And the with-cannoli option pointing at the absolute path to the Cannoli build.

Then the build:

make -j48

And then the example symbolizer, which is the piece to look at to understand what the output looks like.

You start the symbolizer in one terminal:

cd examples/symbolizer
cargo run --release

And run the program under QEMU in another:

cd examples/symbolizer
</path/to/qemu>/build/qemu-mipsel -cannoli </path/to/cannoli>/target/release/<your jitter so>.so ./example_app

The qemu-mipsel binary with the -cannoli flag and the path to the compiled jitter shared object, running the example app. The result is an image showing memory accesses and PC executions side by side, which is the closest thing in the repository to a picture of the output.

Nix takes the QEMU patching out of your hands

There are two escape hatches from the manual path, and both are worth knowing about.

The first is the Nix flake. If you have Nix, getting a build of QEMU with the Cannoli patches is a single command:

nix --experimental-features 'nix-command flakes' build

That is a real saving, because the manual path is four steps that each involve getting a commit hash right.

The second is pre-built binaries. If desired, you may also skip building and download pre-built binaries, and the prompts that Nix shows for this are given:

do you want to allow configuration setting 'extra-substituters' to be set to 'https://cannoli.cachix.org' (y/N)? y
do you want to allow configuration setting 'extra-trusted-public-keys' to be set to 'cannoli.cachix.org-1:nFKY7lRczFkkHacy6/OlfmpOU22MeEiDo90YV0QkVoQ=' (y/N)? y

The supported emulators end up in ./result/bin after a few minutes of compiling.

There is also a nix devshell, which you can use to populate a temporary shell with Cannoli and the supported Rust toolchain, for development of applications that consume Cannoli traces:

nix --experimental-features 'nix-command flakes' develop

So the three ways to get the pieces are: build it yourself with cargo and a patched QEMU, build it with the flake and get pre-built binaries from the binary cache, or use the devshell for development while the binaries come from the cache.

The repository layout reflects that this is a project with two halves. There is a cannoli/ directory for the main library, a jitter/ directory for the JIT hook library, and a jitter_always/ directory with a basic example that hooks everything. There is a mempipe/ directory, which is the mechanism for the event stream, and a qemu_patches.patch file at the root. The extensions/ directory suggests the hook interface is designed to be extended, and there is a rust-toolchain.toml and a .cargo/ directory pinning the toolchain.

The examples are numerous and they map onto the uses. There is a symbolizer, a coverage example, a benchmark and a benchmark_graph, a backtrace, a regtrace for register tracing, a visualization example and a ci_runner.py, which is a script rather than a Rust program and suggests the library is used from Python as well.

The jitter hook is where you decide what gets traced

The user experience section explains what you actually write, and it is a small amount of code.

The shared library loaded into QEMU is called the Cannoli Jitter. Using this library expects two basic callbacks to be implemented, such that QEMU knows when to hook, and how to hook, certain operations. This is the filter mechanism that prevents JIT code from being produced in the first place if you do not want to hook literally everything.

use jitter::HookType;

/// Called before an instruction is lifted in QEMU.
///
/// The `HookType` dictates the type of hook used for the instruction, and may
/// be `Never`, `Always`, and `Once`
///
/// This may be called from multiple threads
#[no_mangle]
fn hook_inst(_pc: u64, _branch: bool) -> HookType {
    HookType::Always
}

/// Called when a memory access is being lifted in QEMU. Returning `true` will
/// cause the memory access to generate events in the trace buffer.
///
/// This may be called from multiple threads
#[no_mangle]
fn hook_mem(_pc: u64, _write: bool, _size: usize) -> bool {
    true
}

The two callbacks are hook_inst, called before an instruction is lifted in QEMU, and hook_mem, called when a memory access is being lifted.

The HookType returned from hook_inst dictates the type of hook used for the instruction, and may be Never, Always and Once. That is the interesting type. Never means the instruction is not instrumented at all, so no JIT code is generated for it. Always means it is instrumented every time. Once means it is instrumented the first time and not again, which is how you get one event per code address rather than one per execution of it.

Both callbacks may be called from multiple threads, which is why the return values are simple and the hook types are coarse. A hook that had to track per-thread state would be far more expensive.

For hook_mem the return value is a boolean. Returning true, which is the default, causes the memory access to generate events in the trace buffer. Returning false means no instrumentation is added to the JIT, and thus QEMU runs with full speed emulation.

The performance argument is stated in the same terms. These hooks provide an opportunity for a user to decide whether or not a given instruction or memory access should be hooked.

And the API is invoked when QEMU lifts target instructions, which the documentation notes is the core operation of the JIT. That is why the filtering is possible at all, and why the hook decisions happen at lift time rather than at execution time. A decision made at lift time costs nothing at run time, because the run time behaviour is already baked into the emitted code.

The coverage example shows the cost being self-cancelling

The coverage example is the most clever piece of the project, and it demonstrates the point about not instrumenting what you do not need.

Cannoli can be used to get coverage of binary applications for pretty cheap. There is an example provided that uses terrible symbol resolution, but it gives you a rough idea of what you can do.

You build and run the client:

cd cannoli/examples/coverage
cargo run --release

And then invoke QEMU on a binary you want coverage of, using the coverage hooks, with the shared library passed in through an environment variable:

QEMU_CANNOLI=cannoli/target/release/libcoverage.so qemu/build/qemu-x86_64 /usr/bin/vlc

The QEMU_CANNOLI variable is how the library is located, and it is set to the compiled libcoverage.so path.

Then the part that makes it cheap. This should work even for large, many-threaded applications, and the coverage is self-silencing, meaning it will disable reporting of coverage in the JIT by patching itself out once it executes for the first time. You might get events in the future for the same callbacks due to re-JITting of the same code, but it is just meant to be a major filter to cut down on the traffic that you would otherwise get with full tracing.

So the coverage tool does not record every instruction and then work out which ones are new. It patches the instrumentation out of the JIT the first time it runs, and only the first execution of each code path produces an event. The trace is then a list of the code paths that were entered, which for coverage is exactly what you want, and the cost of a path that runs a million times is one event rather than a million.

The re-JIT caveat is honest. The documentation notes you might get events again for the same callbacks due to re-JITting of the same code, because QEMU can re-translate a block, and the result is duplicate events for the same address.

That is the trade the whole library makes. You do not get a complete, ordered trace of every instruction. You get the events you asked for, filtered at the point where filtering is free.

Editorial conclusion

Adopt Cannoli if you are analysing binaries under qemu-user and need both instruction-level and memory-level events at a rate that rules out tracing inside the emulator. Skip it if you need kernel tracing, since this is a user-mode emulator tool, and if you cannot maintain a patched QEMU, because the whole design depends on a patch to the emulator's JIT. Verify first which QEMU commit you patch, since the documentation pins a specific hash and the tool is tied to the internals of that version, that your analysis can live in a separate process, and that you understand the jitter hook model, since deciding what to instrument is what makes the throughput possible.

Frequently asked questions

Official sources

1. Issues
2. License: GPL-2.0
3. MarginResearch/cannoli on GitHub
4. README