ncnn: A Dependency-Free Inference Runtime for Phones, Browsers and Edge Boards

The problem ncnn solves: a model file that runs without a runtime

Most deep learning deployment stories assume you can ship a Python interpreter, a set of shared libraries and a GPU driver stack alongside the model. On a phone, a set-top box or a microcontroller-class board, that assumption fails. ncnn takes the opposite position. The README describes it as a framework with no third-party runtime dependencies, and the whole design follows from that constraint: the network is loaded from two files, a .param graph description and a .bin weight blob, and executed by a C++ library you link directly into your application.

The intended audience is narrow and specific. It is the engineer who has a trained model and needs it to run inside an existing native app. The README names Tencent's own deployments (QQ, Qzone, WeChat, Pitu) as evidence that this path is exercised in production, though it gives no numbers about model size, latency or device coverage. What the repository does show is breadth of build support: Linux, Windows, macOS, Raspberry Pi 3 and Pi4, POWER, Android, NVIDIA Jetson, iOS, WebAssembly, AllWinner D1 and Loongson 2K1000 are all listed in the build wiki. That list is the clearest statement of intent. This is not a framework for a data centre; it is one for whatever silicon is already in the product.

How a PyTorch model becomes a .param and a .bin

The documented path is PyTorch to pnnx to ncnn, and the README presents it as the recommended route for beginners. pnnx is installed as a Python package with pip3 install pnnx, then called from inside a PyTorch script. The README's example defines a small module with a convolution, a ReLU, a global mean and a linear layer, runs it in eval mode, and calls pnnx.export(model, "model.pt", (x,)) with a dummy tensor of shape 1x3x224x224. The stated output is a pair of files, model.ncnn.param and model.ncnn.bin.

That two-file split is the core architectural decision. The .param file holds the layer graph and the connections between layers; the .bin file holds the weights. Loading is therefore two separate calls, load_param then load_model, which means the graph can be inspected or edited as text without touching the weight blob. Inference is explicit rather than implicit: you create an extractor from the net, call input with a named blob, then call extract with the output blob name. The README example uses the names in0 and out0, which are assigned during conversion rather than derived from the original PyTorch module. If you rename anything in the PyTorch graph, expect to re-check those names.

Execution is split across two backends. The CPU path is built around SIMD kernels, with ARM NEON and RISC-V among the topics attached to the repository, and a Vulkan path handles GPU execution. The README does not describe how the two are selected or whether a model runs on both simultaneously, so treat backend selection as something to confirm against the wiki rather than something the README answers.

Getting it running: prebuilt archives or a source build

There are two entry points. The releases page publishes dated archives per platform, and the README links them directly. The naming is worth reading carefully, because the variants are not interchangeable. For Android the project ships ncnn-<date>-android-vulkan.zip and ncnn-<date>-android.zip, the latter marked cpuonly. There is also a shared variant of each, ncnn-<date>-android-vulkan-shared.zip and ncnn-<date>-android-shared.zip. HarmonyOS follows the same pattern with harmonyos-vulkan and cpuonly archives, again in static and shared forms. The same release also carries android, harmonyos, ios, macos, linux, windows, webassembly, watchos, tvos and visionos builds, plus a full source zip, ncnn-<date>-full-source.zip.

The distinction that matters most in practice is vulkan versus cpuonly. If you pick the Vulkan archive you are taking on a GPU dependency; if the device or the driver does not cooperate, the cpuonly archive is the fallback. Static versus shared is the usual trade-off: static linking keeps deployment simple, shared linking lets several components share one copy of the library.

Building from source is documented in the project wiki rather than the README, with separate pages for Linux, Windows, macOS, Raspberry Pi, POWER, Android, NVIDIA Jetson, iOS, WebAssembly, AllWinner D1 and Loongson 2K1000. The README also notes a Termux build path for Android. If your target is not on that list, the wiki is the only guidance available here, and cross-compiling for an unlisted architecture is your problem to solve.

The runtime API itself is small. In C++ you include net.h, construct an ncnn::Net, call load_param and load_model with the two file paths, build an ncnn::Mat for the input, create an extractor, feed it with input and read the result with extract. A Python binding mirrors the same sequence: construct ncnn.Net, load the same two files, wrap a NumPy array with ncnn.Mat, and call extract, which returns a status code and the output Mat. The symmetry is deliberate, which makes prototyping in Python and shipping in C++ a realistic split.

Where ncnn stops being the right tool

The first limitation is inherent to the design: ncnn is an inference framework. The README describes it as an inference framework in both English and Chinese, and nothing in the repository suggests training. If your workflow requires fine-tuning on device, or a graph that changes shape at runtime based on input, the static .param model is working against you rather than for you.

The second is the conversion step. Everything documented flows through pnnx, and pnnx is a separate tool with its own release cadence. A model that uses an operator pnnx does not map cleanly will either fail to convert or convert into something that behaves differently from the PyTorch original. The README does not publish a supported-operator list or a conversion success rate, so the only way to know is to run your own model through it and compare outputs numerically. That comparison is not optional; it is the actual acceptance test.

The third is the Vulkan backend. Shipping the vulkan archive means your application now depends on a working Vulkan driver on the target device. On Android that varies by vendor and by OS version. The README does not describe a fallback policy, so you should not assume that a failed GPU initialisation quietly degrades to CPU. Test on the actual hardware you intend to support, not on a desktop emulator.

Finally, the release naming is a maintenance hazard in itself. Archives are dated rather than semantically versioned (20250916, 20260113, 20260526), so upgrading means diffing dates and reading release notes rather than reasoning about a major version number. There is no indication in the supplied material of a long-term support branch or an API stability guarantee.

How it compares to ONNX Runtime and TFLite

ONNX Runtime takes a different position on the same problem. It defines a portable intermediate format and then builds execution providers around it, so the model is the contract and the runtime is pluggable. ncnn inverts this: the runtime is the fixed thing you link, and pnnx is the adapter that reshapes your model to fit it. If your deployment spans several runtimes and you want one artifact to feed all of them, the ONNX route is more natural. If you have exactly one target and want the smallest possible dependency surface, ncnn's approach removes the layer of indirection entirely.

TensorFlow Lite is closer in spirit, since it also targets mobile with a compact runtime. The practical difference visible here is backend choice and build breadth. ncnn ships a Vulkan GPU path alongside the CPU path and publishes archives for WebAssembly, watchOS, tvOS and visionOS in the same release as Android and iOS. The README does not compare performance against either alternative, and this article will not either, because no benchmark numbers are supplied. The honest comparison is architectural: ncnn's selling point is that the .param and .bin files plus a linked C++ library are the entire deployment, with pnnx as the only tool in the chain.

Licence and upgrade cost

The README displays a BSD 3-Clause badge linking to LICENSE.txt, but the repository metadata reports the licence as NOASSERTION, which means an automated classifier could not match the file to a known SPDX identifier. Those two signals disagree. Before you ship ncnn inside a commercial product, read LICENSE.txt directly and have whoever handles licensing at your organisation confirm the terms; this article cannot and does not give legal advice.

The upgrade cost is real but bounded. Model files are produced by pnnx, so a pnnx upgrade can change what the .param graph looks like, and the runtime and the converter need to move together. Because releases are dated, you cannot rely on a version number to tell you whether an upgrade is a patch or a breaking change. A workable discipline is to pin a specific dated archive per platform, keep the pnnx version that produced your model files recorded alongside them, and re-run your numerical comparison after any change to either side. That is more bookkeeping than a semantic-versioned library demands, and it is the price of the two-file model format.

Who should adopt ncnn, and what to check first

Adopt ncnn if you are embedding a fixed, already-trained model into a native application and the deployment target is a phone, a browser via WebAssembly, or one of the boards the build wiki covers. The absence of third-party runtime dependencies is the whole argument, and it holds up: two files, a linked library, and an API that is a handful of calls in either C++ or Python.

Do not adopt it if you need training, dynamic graphs, or a workflow where you iterate on the model inside a Python notebook and expect the same artifact to run everywhere. Do not adopt it if your target architecture is not in the build list and you are not prepared to write the cross-compilation toolchain yourself. And do not adopt the vulkan archive without confirming that your target devices have working Vulkan drivers.

Three checks before you commit. First, run your own model through pnnx and compare the ncnn output against the PyTorch output numerically; the README's example model is a four-layer toy and tells you nothing about your operator coverage. Second, confirm the blob names that pnnx assigns, since the README's in0 and out0 are conversion artifacts and your input and extract calls must match them exactly. Third, download the specific dated archive for your platform and ABI (vulkan or cpuonly, static or shared) and confirm it links into a minimal build before you restructure anything around it. If those three pass, the rest of the integration is small.

Editorial conclusion

Adopt ncnn if you are shipping a fixed model inside an Android, iOS, HarmonyOS or embedded binary and cannot afford to drag a Python or CUDA runtime along. Do not adopt it if you need training, dynamic control flow, or a workflow where the model is edited and re-run interactively. Before committing, verify three things on your own target: that pnnx converts your operator set without falling back to unsupported ops, that the Vulkan backend actually initialises on your device rather than silently running on CPU, and that the release archive you download matches your ABI, since the project publishes separate vulkan, cpuonly, static and shared variants per platform.

Official sources

1. Issues
2. README
3. Releases
4. Tencent/ncnn on GitHub