Nanoflow: intra-device parallelism for LLM serving, and what it costs you

The problem Nanoflow attacks: idle silicon inside a single GPU

Most LLM serving stacks scale out before they scale in. Tensor parallelism, pipeline parallelism and data parallelism all spread a model or a request stream across devices, and each of them assumes the per-device pipeline is already efficient. Nanoflow's premise, stated in the abstract, is that this assumption is false: existing methods "do not consider overlapping the utilization of different resources within a single device, leading to underutilization and sub-optimal performance." During a decode step, the GEMMs are compute-bound, attention and KV-cache traffic are memory-bound, and any multi-GPU communication is network-bound. Executed sequentially, each phase leaves the other execution units waiting. The target user is not someone serving a single model to a small team. It is an operator with a cluster of A100 80GB SXM GPUs, a throughput target measured in tokens per second per GPU, and enough engineering time to rebuild the inference stack. The README frames the motivation at that scale: "tens of thousands of GPUs continuously serve hundreds of millions of users." If your deployment is one or two GPUs behind an application, the overlap Nanoflow buys is real but small in absolute terms, and the integration cost is not.

Nano-batching and the device-level pipeline

The mechanism is a re-granularisation of the batch. Conventional continuous batching forms a batch per iteration, then runs the whole batch through each stage. Nanoflow splits requests at the granularity of operations, which the README calls nano-batching, and the abstract says it "breaks the dependency of sequential operations in LLM inference and enables overlapping them." Once the dependency is broken, a device-level pipeline with execution unit scheduling can co-schedule a compute-heavy nano-batch alongside a memory-heavy one, so the compute-bound work stays on the critical path while memory and network traffic fill the gaps. The README's own summary of the payoff is blunt: "Such overlapping leaves compute-bound operations on the critical path and boost the resource utilization." The second half of the design is the CPU side. With the GPU better utilised, the CPU work of KV-cache management, batch formation and selecting retired requests becomes a visible fraction of iteration time, which the README puts at more than 10 percent. Nanoflow therefore runs an asynchronous control flow: at iteration i it makes batching decisions and allocates KV-cache entries for iteration i+1 before iteration i finishes, launches i+1 without checking for end-of-sequence tokens produced in i, and retires completed requests at i+2. That is a two-iteration lag between a request finishing and the scheduler noticing, which is a deliberate trade of scheduling accuracy for pipeline depth.

KV-cache offload to SSD, and the bandwidth arithmetic behind it

Multi-round conversations reuse prefixes, and recomputing them wastes the very GPU time Nanoflow is trying to protect. The README describes an eager offload path: when a request finishes, its KV-cache is copied to host and then to SSD, layer by layer, in parallel with the inference operations still running on the GPU. The sizing argument is given explicitly for LLaMA2-70B: offloading requires about 5GB/s, while "a single SSD can reach 3GB/s." That arithmetic implies more than one SSD, or a faster device, to keep up with a 70B model's retirement rate. It is also a design point that assumes local NVMe rather than network storage, since the copy is described as running alongside on-the-fly inference. The README presents this as a capability of the framework rather than a documented configuration surface, so treat the offload path as something to confirm in the source before you plan a deployment around it.

Building it: a C++ backend, a Python front end, and three kernel libraries

The repository is roughly 4K lines split into a C++ backend and a Python demo front end, and the default branch is Nanoflow-python. The README states that the codebase "contains necessary scripts for environment setup and experiment reproduction," and the 2024/09 news entry says experiment scripts to reproduce the evaluation results were released alongside support for Llama2 70B, Llama3 70B, Llama3.1 70B, Llama3 8B, Llama3.1 8B and Qwen2 72B. What the supplied material does not give is the literal command sequence or the configuration keys. There is no install snippet, no CMake invocation, no YAML or JSON schema in the README text provided, so any build instructions here would be invented. The concrete dependency facts you can act on are the three kernel libraries the README names: CUTLASS for GEMM, FlashInfer for attention, and MSCCL++ for network. Each is a separate build with its own CUDA and compiler requirements, and MSCCL++ in particular pulls in NCCL-adjacent assumptions. Plan on a container with a pinned CUDA toolkit, and read the setup scripts in the repository rather than guessing at flags. The absence of releases means you are tracking a branch, not a version.

The failure modes the design accepts

The asynchronous control flow is the clearest trade-off. Because Nanoflow launches iteration i+1 without inspecting the EOS tokens from iteration i, a request that finished is only retired at i+2. Under a long steady stream of similar-length requests this is invisible. Under bursty traffic with highly variable output lengths, the scheduler is making batch and KV-cache decisions on stale information for two iterations, and the extra slots held by finished requests are capacity you cannot use. The second limitation is hardware. Every benchmark in the README is on A100 80GB SXM, and the overlap depends on that device's ratio of compute to memory bandwidth. There is no evidence in the supplied material about behaviour on consumer GPUs, on H100, or on any accelerator without NVLink-class interconnect, and intra-device parallelism does not remove the need for the inter-device parallelism that MSCCL++ provides. Third, the baseline comparison is deliberately narrow: the README notes that "all frameworks turn off specific optimizations like quantization, speculative decoding, prefix cache, etc." That is the right way to isolate the scheduling contribution, but it means the headline comparison is not the comparison you would run in production, where those optimisations are usually on. Finally, the language mix matters. The primary language is listed as Jupyter Notebook, yet the README describes a C++ backend and a Python front end. Expect the notebook content to be experiment and evaluation code, not the serving path, and budget accordingly.

How it differs from vLLM and TensorRT-LLM

The README benchmarks Nanoflow against vLLM v0.5.3, Deepspeed-FastGen v0.2.3 and TensorRT-LLM v0.8.0, and claims up to a 1.91x throughput improvement over TensorRT-LLM. The interesting difference is not the number, it is the axis. vLLM's central abstraction is PagedAttention and a block manager that makes KV-cache memory the scheduling resource; its optimisations are mostly about fitting more concurrent sequences into the same memory and about prefix reuse. TensorRT-LLM takes a compiled-graph route: you build an engine for a specific model, batch shape and GPU, and the runtime executes that plan. Nanoflow changes neither the memory manager nor the compilation model in its core claim. It changes when operations execute relative to each other, using nano-batching and execution unit scheduling to fill idle units, and it adds an async CPU loop so the scheduler never blocks the GPU. That means the three are not mutually exclusive in principle, but in practice you pick one runtime. If your bottleneck is KV-cache capacity, vLLM's block manager is the more direct answer. If your bottleneck is kernel launch overhead and you can tolerate engine rebuilds, TensorRT-LLM is the more mature path. Nanoflow is the right answer when your bottleneck is that compute, memory and network phases are serialised inside the device and you have the engineering capacity to exploit the overlap.

Maintenance, licence and what to pin

There are no retrieved releases, so there is no version to pin and no changelog to read. The last push recorded is 2026-03-29 on the Nanoflow-python branch, which suggests the project is still being touched, but activity on a branch is not the same as a supported release. The licence is listed as unknown in the repository metadata, and the supplied README contains no licence section. Under most interpretations that means no granted rights by default, which is a blocker for commercial deployment and for redistribution inside a product. This is a factual observation about the repository state, not legal advice; if you need to ship Nanoflow, get the licence question resolved by someone qualified before you invest engineering time. On maintenance cost, the practical burden is the dependency triangle. CUTLASS, FlashInfer and MSCCL++ each move independently, and a CUDA toolkit upgrade can force a rebuild of all three plus the Nanoflow backend. Budget for a pinned container image rather than a pip install, and treat every dependency bump as a validation exercise against the reproduction scripts. The 4K-line codebase is small enough to read, which is the one genuine advantage of adopting research code: you can audit the scheduling logic instead of trusting it.

Editorial conclusion

Adopt Nanoflow only if you are running Llama2-70B, Llama3 70B, Llama3.1 70B, Llama3 8B, Llama3.1 8B or Qwen2 72B on A100-class hardware, you are willing to build a CUTLASS, FlashInfer and MSCCL++ stack from source, and your workload is throughput-bound rather than latency-bound. Do not adopt it if you need a supported release, a documented licence, or a drop-in OpenAI-compatible endpoint; the repository has no releases and no licence file in the supplied material, which alone blocks most commercial use. Verify three things before writing any code: whether a LICENSE file exists on the Nanoflow-python branch, whether the release scripts reproduce the paper's 8xA100 numbers on your hardware, and whether the eager SSD KV-cache offload path is present and working in the branch you clone, since the README describes it as a design feature rather than a documented runtime option.

Official sources

1. efeslab/Nanoflow on GitHub
2. Issues
3. Project website
4. README