candle-vllm: a Rust inference server for local LLMs, read from the repository
Efficent platform for inference and serving local LLMs including an OpenAI compatible API server.
At a glance
- What is it?
- candle-vllm is an MIT-licensed Rust project that serves local language models behind an OpenAI-compatible HTTP API, with CUDA and Metal backends, paged KV cache, and a long list of quantization paths. The interesting part is not the feature list but where the project draws its portability line.
- Who is it for?
- Adopt candle-vllm if you want a single Rust binary that serves a local model over an OpenAI-compatible endpoint and you are willing to treat the build features as part of your deployment contract. Do not adopt it if you need a formal compatibility matrix for every model family, or if your hardware sits outside the CUDA and Metal paths the README names.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 9 days ago.
- What is it written in?
- Mainly Rust, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap candle-vllm is trying to fill
Serving a model locally usually means choosing between a Python stack that pulls in a large dependency tree and a hand-written script that speaks no standard protocol. candle-vllm sits in the middle. The README describes it as an "Efficient, easy-to-use platform for inference and serving local LLMs including an OpenAI compatible API server", and the OpenAI-compatible part is the load-bearing phrase. If your client already speaks that API, swapping the base URL is the whole integration.
The audience is narrower than the tagline suggests. This is for people who have a GPU or an Apple Silicon machine, a model they want to keep on their own hardware, and a reason to prefer a compiled binary over a Python process. The README's own examples lean toward large quantized models: GLM-5.2-FP8 across eight devices, a 122B GGUF with tensor parallelism over two, Qwen3.5 MTP speculative decoding. Those are not laptop workloads. The Apple Silicon demo is a counterweight, showing a Qwen3-8B at Q2K on an M4 with 16GB of unified memory, but the center of gravity is multi-GPU serving.
What the architecture actually looks like from the outside
The README does not include a design document, so the mechanism has to be read off the feature list and the flags. Several things are visible. The server keeps a paged KV cache, the same PagedAttention idea popularized elsewhere, and it batches continuously, meaning requests that arrive while others are decoding get folded into the running batch rather than waiting for a clean slot. Prefix caching is listed separately, so repeated prompt prefixes can be reused across requests.
Chunked prefilling is on by default with a chunk size of 8K. That is a deliberate memory trade: a long prompt is processed in slices so the prefill does not spike activation memory, at the cost of some scheduling overhead. CUDA Graphs are supported, which usually means decode steps are captured and replayed to cut kernel launch cost. The README also mentions a trait-based architecture for adding new model pipelines, which is the extension point if you need a family that is not covered.
On quantization the project is unusually broad. It lists in-situ quantization with an --isq flag, GPTQ and Marlin 4-bit formats, AWQ including pack-quantized MoE, MXFP4 and NVFP4, block-wise FP8 for SM90 and above, FP8 KV cache on both CUDA and Metal, and TurboQuant KV cache in turbo8, turbo4 and turbo3 variants with native flash attention kernels. The README claims TurboQuant's 2 to 4 bit KV cache "extends context up to 4.7x with minimal quality loss". That 4.7x figure is the project's own claim and the phrase "minimal quality loss" is not quantified anywhere in the material, so treat it as a starting hypothesis to test on your workload rather than a measured result.
Getting a server up: the three documented paths
The README gives three install routes. The one-line script is curl -sSL https://ericlbuehler.github.io/candle-vllm/install.sh | bash, which the README says yields a DEB package or a binary. Building from source is a cargo install with feature flags: cargo install --features cuda,nccl,flashinfer,cutlass --path . for CUDA, and cargo install --features metal --path . for macOS. The README notes that flashinfer and cutlass should be removed for sm_70 and sm_75, which tells you the feature set is tied to your GPU generation. There is also a Docker path via ./build_docker.sh, which takes the feature string, an SM version and a CUDA version, for example ./build_docker.sh "cuda,nccl,flashinfer,cutlass" sm_90 13.0.0.
Running is a single command. candle-vllm --m Qwen/Qwen3.6-27B-FP8 --ui-server pulls a Hugging Face model ID. For a GGUF you add a file selector: candle-vllm --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_S --d 0,1 --ui-server. The --d flag lists GPU indices, so --d 0,1,2,3,4,5,6,7 spreads work across eight devices. Local paths work too, either a safetensors directory or a GGUF file, and the README says a directory containing GGUF files is auto-detected. Other flags worth noting: --isq q4k for on-the-fly quantization, --mtp 2 for Qwen3.5 speculative decoding with two draft tokens per step, and --yarn-scaling-factor 4.0 for a manual RoPE scaling override.
The API listens on http://localhost:2000 by default. Adding --ui-server also starts a ChatGPT-style web UI, and the README states that the UI server uses the API port minus one, so an API on 2000 puts the UI on 1999. That off-by-one is worth knowing before you containerize: two ports to expose, and a collision if 1999 is already taken.
Where the portability claim breaks down
The README's cross-platform row says CUDA on Linux and Metal on macOS, "same codebase, same API". That is accurate as far as it goes, but the feature lists are not symmetric. FlashInfer, cutlass, NCCL, CUDA Graphs, block-wise FP8 for SM90 and above, and multi-node TCP inference all sit on the CUDA side. Multi-GPU tensor parallelism in both multi-process and multi-threaded modes is a CUDA story. On Metal you get the API server and FP8 KV cache, and the README's Apple Silicon demo is a single 8B model at Q2K on 16GB of unified memory.
The deeper limitation is model coverage, and the README is honest about it in a way that is easy to miss. The performance table has TBD in the BF16 column for DeepSeek V2/V3/V3.2/R1, GLM4.7 Flash, LLama4, MiniMax-M2.5/M2.7 and GLM-5.2. For DeepSeek the only figure given is roughly 20 tokens per second for an AWQ 671B with tp=8 and offloading. Those TBD cells mean the maintainers have not published a number, not that the model fails. But if your chosen family is in that list, you are adopting ahead of the published evidence.
The second failure mode is build configuration. Because the feature flags encode your GPU generation, a binary built for the wrong SM version is a runtime problem, not a compile-time one in every case. The README's instruction to drop flashinfer and cutlass for sm_70 and sm_75 is the only compatibility guidance in the material. There is no matrix mapping model families to required features. Expect to read the flag list and reason about it yourself.
How this differs from vLLM and llama.cpp
The name invites the comparison, so it is worth being precise. vLLM is the Python project that established PagedAttention and continuous batching as the default vocabulary for this kind of server, and it carries a large ecosystem of model support and a Python extension surface. candle-vllm reimplements that serving model in Rust on top of the Candle tensor library. The practical difference is the deployment artifact: a compiled binary with an explicit feature set instead of a Python environment with compiled kernels inside it. That cuts the runtime dependency surface and makes the build the place where hardware assumptions are declared. It also means adding support for a new architecture means writing Rust against the project's traits, not dropping in a Python module.
llama.cpp is the other reference point, and the split is different again. llama.cpp is built around GGUF and CPU-first execution with GPU offload as an option, which is why it runs on hardware this project does not mention at all. candle-vllm treats GGUF as one input format among several (the README shows --f selecting a quant type inside a GGUF repo) and assumes an accelerator is present. If your target is a CPU-only box or an unusual accelerator, llama.cpp is the better fit and candle-vllm is the wrong tool. If your target is a CUDA machine serving an OpenAI-shaped API to existing clients, candle-vllm's build flags and paged cache are aimed squarely at that.
Maintenance, releases and the MIT licence
The release cadence visible in the material is fast: v0.8.9 in late July 2026, v0.9.0 later that month, v0.9.1 in late August, with the last push to master in early September 2026. Three releases in roughly six weeks on a project of this scope implies active development, and it also implies upgrade cost. Flags and feature names are the kind of surface that moves during rapid iteration, so pinning a version and reading the release notes before bumping is the sane posture. The README does not describe a deprecation policy or a stability guarantee for the CLI, so assume none.
The licence is MIT. That is permissive: it allows use, modification and redistribution with the licence and copyright notice preserved, and it comes without warranty. Nothing in the material suggests a separate commercial licence or a usage restriction, but the repository also does not document the licences of the model weights you would load, and those are governed by their own terms. FP8 and NVFP4 checkpoints from model vendors frequently carry their own conditions. That is a question for whoever handles licensing on your side, not something this article can settle.
Who should run this, and what to confirm first
The case for candle-vllm is strongest when you want one binary, an OpenAI-compatible endpoint, and a CUDA machine with enough VRAM for a quantized model. The documented flag set covers the serving knobs that matter: device selection with --d, quantized weights via --f or --isq, speculative decoding with --mtp, and a bundled web UI behind --ui-server. If you are already pointing clients at an OpenAI-shaped base URL, the migration is a port change.
The case is weak in three situations. You need CPU-only inference, which the README does not claim to support. Your model family sits in the TBD rows of the performance table, so you would be the one producing the first number. Or you need a documented compatibility matrix before you commit, because the material gives you a feature list and one note about sm_70 and sm_75, not a table.
What to verify before you deploy: run the model you actually intend to serve, at the quantization you actually intend to use, and measure decode throughput yourself rather than reading the Hopper 80G table. Confirm your SM version against the build features, since the README's only guidance there is to drop flashinfer and cutlass below sm_80. And check that port 1999 is free if you plan to use --ui-server alongside an API on 2000.
Editorial conclusion
Adopt candle-vllm if you want a single Rust binary that serves a local model over an OpenAI-compatible endpoint and you are willing to treat the build features as part of your deployment contract. Do not adopt it if you need a formal compatibility matrix for every model family, or if your hardware sits outside the CUDA and Metal paths the README names. Before committing, verify three things against your own machine: that the model you intend to serve appears in the README's performance table with a real number rather than TBD, that your GPU's SM version is covered by the build features you pass to cargo install, and that the UI port (API port minus one) does not collide with anything already bound on the host.
Community notes