XNNPACK: the operator layer most mobile inference actually runs on

A library underneath frameworks, not a tool you run

The positioning sentence in the README is unusually direct, and it is worth quoting the logic of it rather than the phrasing. XNNPACK is described as a highly optimized solution for neural network inference on ARM, x86, WebAssembly and RISC-V. It is then stated plainly that the library is not intended for direct use by deep learning practitioners and researchers. What it provides instead are low-level performance primitives for accelerating high-level machine learning frameworks.

The frameworks named in that paragraph are the real user base: TensorFlow Lite, TensorFlow.js, PyTorch, ONNX Runtime, ExecuTorch and MediaPipe. Six projects, each with its own graph representation, its own memory planner and its own deployment story, all converging on the same set of kernels because those kernels are good and because writing your own is a poor use of a framework team's time.

This has an important consequence for anyone evaluating it. You will probably never choose XNNPACK. You will choose TensorFlow Lite or ONNX Runtime, notice that its CPU execution provider is XNNPACK, and inherit the decision. The questions that matter are therefore about coverage and portability rather than about API design, and the README answers the coverage question well and the rest not at all.

Where the operator list stops

The README enumerates the operators, and the list is long enough to be useful precisely because of where it ends. Convolution is there in several forms: 2D convolution including grouped and depthwise variants, 2D deconvolution also known as transposed convolution, 2D average pooling, 2D max pooling, 2D argmax pooling which returns indices alongside the max values, 2D unpooling, 2D bilinear resize, and 2D depth-to-space which is pixel shuffle.

The elementwise list is broad in a way that suggests these are the operations that showed up repeatedly in real mobile models. Add, subtract, divide, maximum, minimum, multiply and squared difference all carry a broadcasting note, and add is qualified as two inputs only, which is a small but real constraint. Then there is fully connected, channel shuffle, global average pooling, transpose and copy.

The activation and conversion entries are the part that tells you how much quantization awareness lives in the library. Clamp covers ReLU and ReLU6, convert explicitly handles fixed-point and half-precision quantization and dequantization, and banker's rounding, ceiling, floor and truncation each have their own entry. PReLU, ELU, leaky ReLU, hard-swish, sigmoid, softmax, tanh, square, abs and negate fill out the rest.

What is absent matters as much. There is no matrix multiply, no attention operation, no transformer-specific primitive, and no embedding lookup in that list. For a library that predates the current wave of attention-based mobile models, that boundary is not a flaw so much as a description of what it was built for.

NHWC with a free channel slice

One design decision in the README is more consequential than the operator list, and it is easy to skim past. Every operator supports NHWC layout, and additionally allows a custom stride along the channel dimension. The consequence spelled out is that operators can consume a subset of channels in the input tensor and produce a subset of channels in the output tensor.

That sounds like an implementation detail and behaves like a compiler optimisation. If an operator can read only the channels it needs and write only the channels it produces, then splitting a wide tensor into two narrow ones, running different operations on each, and concatenating the results costs nothing in memory traffic. Those split and concatenate nodes, which are extremely common in networks designed for cheap inference, disappear from the critical path.

It also means channel counts no longer have to be multiples of anything the hardware likes. The SIMD width on a given target does not constrain your graph, because the library handles the edges. This is the kind of decision that looks like nothing from outside the library and saves a measurable fraction of runtime inside every model that uses it.

Reading the published benchmark tables honestly

The README publishes two sets of numbers, and both carry conditions that matter more than the figures. The mobile figures are latency in milliseconds for three generations of MobileNet on three generations of Pixel phone, given separately for single-threaded and multi-threaded execution, where the thread count is however many big cores the device has. The single-threaded table runs from 82 ms for MobileNet v1 on a Pixel down to 12 ms for MobileNet v3 Small on a Pixel, and the multi-threaded table drops the first of those to 43 ms while the smallest model goes to 7 ms.

The conditions line is the honest part. The figures were benchmarked on 2020-03-27 using the repository's own end-to-end benchmark harness with a minimum time setting, on an Android ARM64 build made with the NDK at r21, and, importantly, using models with randomized weights and inputs rather than real ones.

The build command that produced the binary is given as well, which is unusual and helpful:

bazel build -c opt --config android_arm64 :end2end_bench

The Raspberry Pi table is newer, from 2022-02-08, and covers four boards plus an ARM64 variant of the Pi 4. The spread is dramatic: MobileNet v1 runs in 3919 ms on a Pi Zero W and 72 ms on a Pi 4, while the quantized versions are consistently faster, with INT8 MobileNet v1 at 2589 ms and 29 ms respectively. That table was produced with CMake and a local build script rather than Bazel.

./scripts/build-local.sh

Two caveats belong with all of it. These are 2020 and 2022 hardware measurements on phone and single-board-computer processors that are now several generations old, and single-threaded numbers say nothing about how the library behaves when an application is also doing other work.

Supported architectures and what sits in the tree

The supported architecture list is long, which is the clearest evidence of the project's actual priorities. ARM64 is covered on Android, iOS, macOS, Linux and Windows. Older ARM is still there: ARMv7 with NEON on Android, and ARMv6 with VFPv2 on Linux. For desktop-class x86 the coverage runs up to AVX512 across Windows, Linux, macOS, Android and the iOS simulator. WebAssembly appears three times, as the MVP baseline, as SIMD, and as relaxed SIMD which is marked experimental. RISC-V is covered at RV32GC and RV64GC. The last entry is the one that surprises people: Hexagon with HVX, which is Qualcomm's DSP architecture.

That Hexagon entry says something about the kind of user this library has. Supporting a DSP means targeting hardware that is not a general-purpose CPU, which is a different set of constraints from picking between ARM and x86 microarchitectures.

The tree backs up the same impression. `bench/` is where the end-to-end benchmark harness the README references lives, `gemm_compiler/` is a generator rather than a hand-written library, `src/` is the operator implementations, `include/` the public headers, and `build_config/` suggests the microkernel selection is configuration driven rather than compiled in. `ynnpack/` points at an external dependency for the graph operator description, `litert/` to an integration path, and `third_party/` to the usual vendored dependencies. There are build definitions for Bazel, CMake and GN side by side, and a `DEPS` file, which is what a project embedded in several other build systems looks like.

Build requirements, licensing and the documentation gap

The stated minimum build requirements are short and worth taking literally: C11, C++17 and Python 3. Python appears because the microkernel sources are generated and the build scripts run them, which is also why a directory named `gen/` exists. There is a `scripts/` directory that includes the local build path used for the Raspberry Pi figures, and a `docker/` directory for reproducible environments.

The license field in the repository metadata is recorded as NOASSERTION, and a `LICENSE` file is present at the root. That combination usually means the licensing terms are specific and are not being expressed in a form an automated scanner recognises, and anyone embedding this in a commercial product should read that file rather than assume.

Documentation is the real gap. Beyond the operator list and the architecture list, the README points to a `doc/` directory and a `BUILD.md`, but the practical guidance lives elsewhere. Notably absent from what is here is any API walkthrough: no example program, no header tour, no discussion of thread pool configuration or weight packing. There are no tagged releases in the visible history either, which for a library consumed through other frameworks is defensible, since the frameworks pin their own versions.

The publications section is the strongest signal of intent. Four papers are listed, including the indirect convolution algorithm and the two-pass softmax algorithm, both by Marat Dukhan, plus fast sparse convolution networks and work on efficient memory management for inference. These are not incremental tuning notes. They are the algorithmic substrate, and a reader who wants to know where the speed comes from should start there rather than in the source.

Editorial conclusion

XNNPACK earns its position by being boring in exactly the way an inference library should be. It implements a defined operator list, it does not want to be called directly by anyone working on models, and its value is realised by frameworks that link it in as a backend. The practical question for any team is not whether the kernels are fast but whether your framework already routes to them, because most of the ones named in the README do. Read the operator coverage list first and see whether your graph fits inside it, then look at the benchmark harness configuration in the README to understand what the published numbers actually measured, and remember they were collected on hardware generations that predate current hardware by a wide margin.

Frequently asked questions

Official sources

1. google/XNNPACK on GitHub
2. Issues
3. README