nndeploy: A Workflow Graph Layer for Deploying AI Models Across Desktop, Mobile and Edge
一款简单易用和高性能的AI部署框架 | An Easy-to-Use and High-Performance AI Deployment Framework
At a glance
- What is it?
- nndeploy is an Apache-2.0 C++ framework that wraps thirteen inference backends behind a visual, JSON-exportable workflow graph. It is a strong fit when one pipeline must reach Windows, Android, iOS and edge devices, and a poor fit when you only ever target one runtime on one machine.
- Who is it for?
- Adopt nndeploy when a single algorithm pipeline has to ship to several of Windows, macOS, Android, iOS, NVIDIA Jetson, Ascend310B or RK hardware, and when the graph is worth expressing once as JSON instead of re-implementing per platform. Do not adopt it for a single-target, single-backend service where an ONNX Runtime or TensorRT session plus your own pre and post processing is less machinery.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 31 days ago.
- What is it written in?
- Mainly C++, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The deployment gap nndeploy is aimed at
Training code produces a model file. Turning that file into something that runs on a Jetson board, an Android phone and a Windows desktop is a separate project, and it is usually a different project each time. Preprocessing gets rewritten in Java, postprocessing gets rewritten in Objective-C, and the inference call itself is buried under three layers of platform glue. nndeploy targets that specific gap. The README frames the problem as "AI 算法在端侧部署" (deploying AI algorithms on the device side) and lists the target surfaces explicitly: desktop (Windows, macOS), mobile (Android, iOS), edge devices (NVIDIA Jetson, Ascend310B, RK) and single-machine servers (RTX series, T4, Ascend310P).
The intended user is an engineer who owns a model and needs it running on more than one of those surfaces without maintaining a separate integration per surface. The README also draws a boundary around a second audience: for models above 10B parameters, including large language models and AIGC generation models, it describes nndeploy as suitable mainly as a visual workflow tool rather than as the inference engine itself. That distinction matters. On a server with an RTX card and a large model, nndeploy is orchestrating; the heavy lifting happens in whatever backend sits underneath.
How the workflow graph and backend abstraction fit together
The core abstraction is a graph of nodes. A node is a unit of work: a preprocessing step, an inference call, a postprocessing step. Edges carry data between them. The README states that over 100 visual nodes have been developed, and that as the node count grows, reuse improves and the cost of deploying the next algorithm falls. That is the central bet of the project: the graph is the artifact you maintain, and the platform-specific work is pushed into the nodes and the backend layer.
Below the graph sits a backend layer. The README lists thirteen inference frameworks with a status column, all marked supported: ONNXRuntime, TensorRT, OpenVINO, MNN, TNN, ncnn, CoreML, AscendCL, RKNN, SNPE, TVM, PyTorch, plus an internal inference submodule documented in docs/zh_cn/introduction/README_INFERENCE.md. The same graph is meant to run against different entries in that table. The README describes the framework as supporting flexible selection of the inference engine, compilation of only the backends you need to reduce dependencies, and an independent running mode for plugging in a custom inference framework.
Execution is not strictly sequential. The README lists serial, pipeline-parallel and task-parallel execution modes, and separately lists zero-copy, memory pooling and memory reuse as the memory strategy. Nodes can be written in Python or C++, including C++/CUDA and Ascend C implementations, and the README states that Python-written and C++/CUDA-written nodes both integrate into the same visual workflow. Custom nodes are the intended extension point when the built-in library does not cover your operator.
Getting a graph running and exporting it
The README does not reproduce a full build transcript, so the exact CMake invocation is not something I can quote here. What it does state is the shape of the workflow: build the framework with the backends you want enabled, build or launch the visual editor, drag nodes to construct the graph, tune parameters with live feedback, then export the graph to JSON. The exported JSON is then loaded through the C++ or Python API. The README names Linux, Windows, macOS and Android as platforms where that exported workflow can be called.
The backend selection is a compile-time decision, not a runtime one. The README says the framework "可按需编译减少依赖" (can be compiled on demand to reduce dependencies). Practically, that means the set of backends you enable at configure time determines what the resulting binary can load. If you leave TensorRT out of the build, a graph containing a TensorRT node has nothing to bind to. This is worth planning before the first build rather than after, because it shapes your CMake flags, your toolchain files and your CI matrix.
The repository ships CI workflows for Linux, Windows, Android, macOS and iOS, visible as badges at the top of the README. Those are the platforms the project itself builds against, which is a more useful signal than a feature table for judging whether your target is actually exercised. Android deployment has a dedicated guide at app/android/README.md. The homepage documentation is the Chinese Read the Docs site at nndeploy-zh.readthedocs.io.
Where the abstraction leaks
The thirteen-backend table is the project's headline claim, and it is also where the most caution is warranted. A backend marked supported means an integration exists. It does not mean every operator your model uses is implemented for that backend, and it does not mean the same graph produces identical output across all thirteen. The README does not publish a per-backend operator coverage matrix or a numerical parity report, and I could not find one in the supplied material. If your model contains an operator that the RKNN or SNPE path does not handle, you will discover it at runtime, not at graph construction time.
Memory reuse and zero-copy are stated as strategies without a documented correctness contract. Reusing buffers across nodes is exactly the kind of optimisation that produces subtle bugs when a node holds a reference longer than the scheduler expects, and the README does not describe how that is prevented. Treat the memory optimisations as something to validate against your own pipeline rather than as a default you can trust on first run.
The 100+ node library is broad but finite. Object detection covers YOLOv5 through YOLOv11 and YOLOx, segmentation covers RBMGv1.4, PPMatting and Segment Anything, OCR covers Paddle OCR, and image generation covers Stable Diffusion 1.5, SDXL and SD3 plus HunyuanDiT. If your model is outside those families, you are writing a custom node, and the ease-of-use argument weakens considerably at that point. The README's own framing for models above 10B parameters, describing nndeploy as suitable as a visual workflow tool, is a hint that the project does not claim to beat a dedicated serving stack on raw large-model throughput.
nndeploy against a direct ONNX Runtime integration
The obvious alternative is to skip the framework and call ONNX Runtime or TensorRT directly. The difference is not performance, it is where the platform-specific code lives. A direct integration means you own the preprocessing, the session setup, the device selection and the postprocessing for each target. On a single Linux server with one GPU, that is a few hundred lines and it is entirely under your control. Nothing in nndeploy removes that work; it relocates it into nodes that someone else may have already written.
The decision turns on how many targets you have. With one target, the direct route wins on dependency count and on debuggability, because there is no graph scheduler between you and the runtime. With four targets, the direct route means four copies of the same preprocessing logic diverging over time, and that is the cost nndeploy is designed to remove. The exported JSON is the mechanism: one graph, loaded by the C++ or Python API on each platform.
There is a middle option worth naming. nndeploy supports an independent running mode for custom inference frameworks, and the README lists PyTorch among the supported backends. That means you can keep your own runtime and still use the graph layer for orchestration, which is a smaller commitment than adopting the full node library.
Licence and the cost of keeping up
nndeploy is Apache-2.0. That permits commercial use, modification and redistribution, and it includes a patent grant. It does not require you to open your own code. The practical implication is that the framework itself is not the constraint; the constraints come from the backends you enable. TensorRT, CoreML, SNPE, RKNN and AscendCL each carry their own licence terms and their own redistribution rules, and compiling them into your binary does not inherit Apache-2.0. If you ship a single binary with several backends linked in, the licence surface is the union of those terms, not the framework's licence. That is a question for your legal team, not for this article.
The maintenance picture from the release history is active but not fast. The most recent release in the supplied material is v3.0.10, dated 2026-04-04, and the repository's last push is 2026-08-15, which suggests ongoing work between releases. The gap between v3.0.8 in December 2025 and v3.0.9 in April 2026 is roughly four months. For a framework that binds to thirteen external runtimes, that cadence matters: every TensorRT, ONNXRuntime and MNN release is a potential break, and you are depending on the maintainers to absorb those changes. Budget for the possibility of pinning backend versions yourself. The 3.x version line also implies API movement across major versions, so pinning a specific tag rather than tracking main is the lower-risk choice for a production pipeline.
Who this is for and what to check before you build
The engineers who benefit are the ones with a multi-target problem and a model family that already has nodes. If your pipeline is YOLO detection, Segment Anything, Paddle OCR or a Stable Diffusion variant, and it has to run on at least two of desktop, mobile and edge, the node library and the JSON export do real work for you. The visual editor also shortens the loop between changing a preprocessing parameter and seeing the result, which is a genuine cost saving during tuning.
The engineers who should look elsewhere are those with a single deployment target and a model outside the covered families. In that case you are paying the dependency and build complexity of a thirteen-backend framework to use one backend and write your own nodes anyway. A direct ONNX Runtime session is less code and fewer moving parts.
Before building, confirm three things against the repository rather than against this article. First, check the backend table for the exact runtime you intend to ship and confirm it is not one of the rows that has fallen behind. Second, check whether your model's operator set is covered by the node library, because a missing operator means a custom C++ or CUDA node and a different effort estimate. Third, decide your backend set at configure time and build only that set, since the README's stated approach of compiling on demand is what keeps the binary and the licence surface manageable. The release tags are the safest anchor: v3.0.10 is the most recent in the supplied material, and the four-month gap before it is the cadence you should expect to absorb.
Editorial conclusion
Adopt nndeploy when a single algorithm pipeline has to ship to several of Windows, macOS, Android, iOS, NVIDIA Jetson, Ascend310B or RK hardware, and when the graph is worth expressing once as JSON instead of re-implementing per platform. Do not adopt it for a single-target, single-backend service where an ONNX Runtime or TensorRT session plus your own pre and post processing is less machinery. Before committing, verify that your exact backend appears in the supported table, that the node you need is not one of the 100+ still missing, and that the Apache-2.0 build you produce is compiled only with the backends you actually intend to ship.
Community notes