vllm.cpp: a C++ port of vLLM's serving core, with GGUF and RadixAttention added
a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features (GGUF, RadixAttention, Cache-aware scheduling, ...)
At a glance
- What is it?
- mudler/vllm.cpp reimplements vLLM's continuous batching and paged KV cache in C++20, then bolts on llama.cpp-style GGUF loading and SGLang-style cache-aware scheduling. It is a fast-moving community port, not a drop-in replacement, and its own README says so.
- Who is it for?
- Adopt vllm.cpp if you want vLLM-style serving semantics inside a C++ process, you are comfortable tracking a project whose README warns that CLI flags and server behavior can change between commits, and you embed through include/vllm.h rather than the command line. Do not adopt it if you need a stable server contract today, if your stack is already Python and you gain nothing from dropping it, or if you depend on an architecture the README does not list as gated.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly C++, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 17, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem vllm.cpp targets, and who it is actually for
vLLM's serving core is the part people want: continuous batching, block-paged KV cache, automatic prefix caching, speculative decoding. Getting that core usually means getting Python, PyTorch, and a dependency tree the README sizes at 140x more to install than the single 66 MiB binary it advertises. vllm.cpp exists to separate the scheduling algorithm from that runtime. It is a from-scratch C++20 engine that reproduces vLLM's serving behaviour while borrowing three other projects' ideas: SGLang's RadixAttention and LPM cache-aware admission, llama.cpp's flat C ABI and direct compute on quantized blocks, and MLX's GEMM on Apple Silicon. The intended reader is someone shipping an inference process inside a C++ application, a container image with a size budget, or an edge box where a Python runtime is the largest thing on disk. The README is explicit that this is a community port, independent and unofficial, not affiliated with the vLLM project, the PyTorch Foundation, or the Linux Foundation.
How the engine is put together: one ABI, four backends, two weight formats
The architecture is a single library behind a flat C ABI declared in include/vllm.h. That header is the stable surface: it is versioned with VLLM_ABI_VERSION, checkable at runtime through vllm_abi_version(), and the README states that it grows by appending fields whose zero value keeps existing behaviour byte-identical, bumping only on an incompatible change. Everything above that header, meaning CLI flags, server routes, and internals, is explicitly not stable. Backends are CUDA, CPU, Metal, and Vulkan, with an MLX GEMM path on Apple Silicon and a native ROCm path for EXL3 that the release notes describe as validated on gfx1151 with zero CPU fallbacks but with discrete AMD validation and competitive performance still unmeasured. Weights arrive as safetensors or GGUF. The README claims 44 registered architectures and separately describes 25+ as gated, which is the honest split: registered means the loader exists, gated means a fixture exercises it. Recent additions run well past text generation into audio and video surfaces, including POST /v1/audio/speech and POST /v1/videos, which is a much wider scope than the tagline suggests and a much larger surface to keep working.
Building vllm.cpp and running a first completion
The README does not print a minimal build transcript, so the reliable path is the release archives. v0.0.2 ships eight server archives covering CPU, CUDA, Vulkan, Metal, and MLX builds from the GitHub Releases page. If you build from source instead, the top level is a CMakeLists.txt project and docs/BUILD.md is the document the README points at, including a ROCm section. The repository also carries a flake.nix and flake.lock for a Nix build, and a docker/ directory. The example you can actually compile is under examples/server, with examples/cli as the smaller entry point.
If you are embedding rather than running a server, the README's instruction is direct: embed through the header. The header exposes a versioned ABI constant and a runtime accessor, so a host program can compare the two before doing anything else.
#include <vllm.h>That include is the only line the README and the repository layout support verbatim. The versioned constant is VLLM_ABI_VERSION and the runtime check is vllm_abi_version(); both are named in the README, which states the header grows by appending fields whose zero value keeps existing behaviour byte-identical and only bumps on an incompatible change. If your build's header and the loaded library disagree, you want to find out at startup rather than at the first malformed request.
On the server side, the release notes name one flag worth knowing before you tune anything: --kv-cache-dtype fp8 stores K/V in 1-byte E4M3 pages. The same note says a fixed memory budget holds twice as many blocks, and that the throughput trade-off is not measured yet. Treat the memory saving as documented and the speed as unknown.
The completion endpoints accept prompt_logprobs, including -1 for the full vocabulary, and the release notes state that unsupported request shapes are rejected with a 400 rather than silently coerced.
Where vllm.cpp is the wrong tool
The README's own warning is the first limitation: internals, CLI flags, and server behaviour can change between commits, and tracking main means expecting breakage. Only the C ABI carries a compatibility promise, and that promise covers the header, not the HTTP surface. If you need a server contract you can pin for a year, this is the wrong project today. The second limitation is the gap between registered and gated. The README describes GLM-5.3 as joining the model registry with its GGUF loader and first-token forward running through the shared expert-streaming path, then states that the real 201.83 GiB artifact has not completed a load and that resumed sparse decoding still needs the indexer side cache. A registered architecture is not a working deployment. The third is that several headline paths carry no speed claim at all: the EXL3 CUDA path supports its 3-bit body and 6-bit output head with the note that no speed claim is available, and the Tenstorrent Qwen runs are described as establishing completion only. The fourth is scope. An engine that also exposes video and audio pipelines has more code paths that can regress than one that serves text. If your workload is plain chat completions on a mainstream architecture, you are carrying that surface for nothing.
vllm.cpp against vLLM and llama.cpp
Against vLLM, the difference is the runtime, not the algorithm. vLLM is the reference implementation of continuous batching and paged KV, and vllm.cpp is a port that cites a pinned vLLM checkout for parity claims; the README's own benchmark badge claims token-exact output and the same throughput on Qwen3.6-27B, and the release notes claim Qwen3-8B MXFP4 decodes 45.45 against 41.94 tok/s. Those are the project's numbers, not independently reproduced here. The practical difference is deployment: vLLM brings Python and PyTorch, vllm.cpp ships a single 66 MiB binary and no Python at inference time. If your team already runs Python services and can absorb the install, the port buys you nothing except image size. Against llama.cpp, the difference runs the other way. llama.cpp is the mature C/C++ inference project whose deployment story vllm.cpp borrows, and vllm.cpp's own README frames it that way. What llama.cpp does not centre is vLLM-style serving: continuous batching as the primary loop, block-paged KV, and RadixAttention-style prefix reuse. If you want a single-user local chat binary, llama.cpp is the safer choice. If you want multi-request serving semantics in C++, that is the gap vllm.cpp is aiming at, and the README's warning about churn is the price of aiming there early.
Licence and the cost of tracking a fast-moving tree
The licence is Apache-2.0, and the repository carries a NOTICE file alongside LICENSE, which is the usual Apache-2.0 pattern for attributing upstream work. That matters here more than in an average project because the engine ports scheduling and serving ideas from vLLM, SGLang, and llama.cpp, and the README maintains a Trademarks section stating that vllm.cpp is not affiliated with, endorsed by, or sponsored by the vLLM project, the PyTorch Foundation, or the Linux Foundation. If you redistribute a binary, read LICENSE and NOTICE together and confirm what attribution your own product needs; that is a question for your counsel, not for this article. The upgrade cost is the part the README makes unusually clear. Because only the C ABI is versioned, a downstream embedder can hold the header constant while the CLI and server move underneath. The corollary is that anyone integrating through the HTTP API has no such anchor and should expect to re-read docs/reference/c-api.md and the release notes at each version bump. The release cadence supports that reading: v0.0.2-alpha, then v0.0.2-alpha1-ci-test, then v0.0.2, all within August 2026, with the last push to main on 2026-09-17.
Editorial conclusion
Adopt vllm.cpp if you want vLLM-style serving semantics inside a C++ process, you are comfortable tracking a project whose README warns that CLI flags and server behavior can change between commits, and you embed through include/vllm.h rather than the command line. Do not adopt it if you need a stable server contract today, if your stack is already Python and you gain nothing from dropping it, or if you depend on an architecture the README does not list as gated. Before committing, check three things: the VLLM_ABI_VERSION your build reports through vllm_abi_version(), whether the specific checkpoint you plan to serve appears in the model registry with a completed load, and which of the eight v0.0.2 release archives matches your accelerator.
Frequently asked questions
Can I use vLLM in C++?
Not vLLM itself, but vllm.cpp is a community C++20 port of vLLM's serving core, covering continuous batching, block-paged KV cache, automatic prefix caching, and speculative decoding. The README states it is independent and unofficial, not affiliated with or endorsed by the vLLM project.
What is vLLM used for?
In vllm.cpp's framing, vLLM is the upstream project whose serving core this port mirrors and measures against: continuous batching, block-paged KV, automatic prefix caching, and speculative decoding. The README describes vllm.cpp as taking that serving core and pairing it with llama.cpp's deployment story and SGLang's scheduling ideas.
Is llama.cpp faster than Ollama?
The README does not cover Ollama or compare its speed with llama.cpp, so this cannot be answered from the repository. What the README does say is that vllm.cpp borrows llama.cpp's deployment story: one library behind a flat C ABI, GGUF straight off the shelf, and compute directly on the quantized blocks.
Does llama.cpp run locally?
The README does not document llama.cpp's deployment modes. It does describe vllm.cpp as running locally across CUDA, CPU, Metal, and Vulkan, with v0.0.2 shipping eight server archives including CPU, CUDA, Vulkan, Metal, and MLX builds.
Community notes