PowerInfer: CPU/GPU LLM Serving Built Around Neuron Activation Locality

The Problem: A 40B Model on One Consumer GPU

Serving a large model locally usually forces a choice. You either quantize aggressively until quality drops, or you accept that the model does not fit in VRAM and watch tokens crawl as weights stream over PCIe. PowerInfer attacks the second problem directly. The README frames the target as a personal computer with a single consumer-grade GPU, and the demo compares PowerInfer against llama.cpp on one RTX 4090 running Falcon(ReLU)-40B-FP16, reporting an 11x speedup under the same hardware and with both engines fully using VRAM. The intended user is someone who owns one gaming-class GPU, not a server, and wants to run a model far larger than that card's memory would normally allow at interactive speed. The project is written in C++ and licensed MIT, which matters if you plan to embed it in a commercial product.

Hot and Cold Neurons: What the Engine Actually Exploits

The design rests on one observation about LLM inference: neuron activation follows a power-law distribution. A small set of neurons, which the project calls hot neurons, fire across nearly every input. The rest, cold neurons, fire depending on the specific prompt. PowerInfer turns that statistical pattern into a memory placement decision. Hot neurons are preloaded onto the GPU so their weights are always resident and fast to reach. Cold neurons are computed on the CPU. Because the cold set is large but infrequently touched per token, the engine avoids moving those weights across the PCIe bus on every step, which is where the bandwidth cost would otherwise sit. Two supporting pieces make this work: adaptive predictors that estimate at runtime which neurons a given input will activate, and neuron-aware sparse operators that skip computation for neurons that are not activated. The README also notes that the abstract's 13.20 tokens/s average and 29.08 tokens/s peak were measured across various LLMs including OPT-175B on a single RTX 4090, roughly 18 percent below a server-grade A100. Those numbers come from the project's own evaluation, not from independent replication.

Sparsity Is the Entry Ticket, Not an Option

This is the constraint that decides whether PowerInfer is useful to you, and it is easy to miss. The mechanism depends on activation locality, and locality is strong in ReLU-based sparse models. The README lists exactly which models work: Falcon-40B, the Llama2 family, the ProSparse Llama2 family, and Bamboo-7B. The project also states plainly that PowerInfer supports inference with llama.cpp's model weights for compatibility, but that there will be no performance gain in that case. Read that sentence twice before benchmarking anything. If you load a dense Llama 2 checkpoint, you get an engine that behaves like a slower llama.cpp, not a faster one. The speedup comes from the sparse checkpoint, and the sparse checkpoints come from a specific line of work: the TurboSparse effort described in the release notes, which the project says sparsified Mistral and Mixtral to nearly 90 percent sparsity, and ProSparse Llama 2 at roughly 90 percent. If no sparse variant of your target model exists, the engine's central optimization has nothing to exploit.

Build Requirements and Platform Boundaries

The README's setup section lists CMake as a prerequisite and points to installation, model weights, and inference as the three getting-started steps, but the supplied text truncates the dependency list mid-sentence, so the full set of build dependencies cannot be confirmed from this material. What is confirmed is the hardware matrix. PowerInfer has been tested on x86-64 CPUs with AVX2 instructions, with or without NVIDIA GPUs, under Linux and under Windows. Apple M-series chips are supported CPU-only on macOS, with an explicit caveat: the project does not optimize for Mac, so the performance improvement there is not significant. A Metal backend for sparse inference on macOS is listed as coming soon, which means it is not available now. AMD support arrived on 2024/5/17 via ROCm, and the release notes mention a 2024 competition to optimize the engine using open-source ROCm/HIP. One practical consequence of the AVX2 requirement: on a CPU without it, the build target does not apply, and there is no fallback path described.

Compatibility with llama.cpp Tooling, and Where It Stops

PowerInfer is a distinct codebase from llama.cpp, but the README says you can use most of the examples/ directory the same way, including server and batched generation. That is a deliberate compatibility choice and it lowers the cost of trying the engine, since existing scripts and serving patterns carry over. It is also a trap if you read it as full parity. Backward compatibility here means the weights load and the example programs run. It does not mean the sparse operators engage. The same sentence that grants compatibility with llama.cpp weights also withdraws the performance benefit, so a working server on llama.cpp weights tells you nothing about PowerInfer's actual advantage. Treat the compatibility layer as a migration convenience for testing prompts and client code, not as evidence about throughput.

The Alternative: llama.cpp and the Dense-Model Path

The natural comparison is llama.cpp, and the README makes it directly, claiming up to 11.69x over llama.cpp while retaining model accuracy. The difference in approach is what matters. llama.cpp runs dense computation and leans on quantization to shrink weights so they fit in memory. PowerInfer keeps the model large and sparse, then splits execution across two processors based on which neurons actually fire. Those are different bets. Quantization degrades every weight slightly; sparsity removes most weights entirely and pays for it with predictor overhead and CPU-side computation. That means llama.cpp remains the better choice when your model has no sparse checkpoint, when your CPU is weak relative to your GPU (the cold-neuron work has to land somewhere), or when you are on Apple silicon, where PowerInfer states it does not optimize. PowerInfer is also not the only sparse-inference effort; the release notes point to PowerInfer-2 for smartphones and a separate SmallThinker framework for on-device inference, both from the same group, which suggests the desktop engine is one point in a broader line rather than the whole story.

Maintenance, Licence, and What to Check Before You Commit

The repository is MIT licensed and not archived, with the last push dated 2026-05-11 and no releases retrieved in the supplied data. MIT terms are permissive and place few obligations on redistribution, but this is a description of the licence identifier, not legal advice, and you should read the LICENSE file and your own counsel's view before shipping. On maintenance cost, the material shows an active project with dated news entries running from the December 2023 release through January 2026, plus a public project kanban for current development focus. What it does not show is a release cadence, since no releases were retrieved. That absence is worth noting: if you depend on versioned artifacts rather than building from main, you are tracking a branch. The upgrade surface is also tied to model availability. Because the speedup depends on sparse checkpoints, a change in which sparse models are published affects you more than a change in the engine's source. Before adopting, confirm AVX2 on your CPU, confirm a sparse checkpoint exists for the model you need, and build from source on your target platform to see whether the toolchain cooperates.

Editorial conclusion

Adopt PowerInfer if you have a single consumer NVIDIA GPU plus an x86-64 CPU with AVX2 and you intend to serve a ReLU-sparse model such as Falcon-40B, ProSparse Llama 2, or Bamboo-7B, because that is the configuration the project targets. Do not adopt it if your model is not sparse, since the README states that running llama.cpp weights through PowerInfer gives no performance gain, and do not expect gains on Apple silicon, where the project states it does not optimize. Before committing, verify three things: that your CPU exposes AVX2, that a sparse checkpoint exists for the model you actually need, and that the build completes on your platform, because the README lists Linux, Windows, and macOS as tested but only Linux and Windows with GPU inference.

Official sources

1. Issues
2. License: MIT
3. README
4. Tiiny-AI/PowerInfer on GitHub