vLLM: a Python and CUDA LLM serving engine whose speed comes from PagedAttention, batching and a long list of kernels

PagedAttention and continuous batching are the core of the throughput claim

The README lists what makes vLLM fast: state-of-the-art serving throughput, efficient management of attention key and value memory with PagedAttention, continuous batching of incoming requests, chunked prefill, and prefix caching. Fast model execution comes from piecewise and full CUDA and HIP graphs, and automatic kernel generation and graph-level transformations use torch.compile.

The design target is many concurrent requests on a shared GPU, where memory used by the key and value cache limits how many requests fit. That is why the project's founding paper is about memory management for LLM serving. The consequence is that vLLM pays off most under concurrent load. For a single user sending one prompt at a time, the batching machinery has little to batch, so the gain is smaller. The README gives no throughput figures in the text, so measure on your own workload.

Quantization and kernel support is broad, but each combination needs checking on your hardware

The quantization list covers FP8, MXFP8 and MXFP4, NVFP4, INT8, INT4, GPTQ and AWQ, GGUF, compressed-tensors, ModelOpt and TorchAO, with more in the documentation. Attention kernels include FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA and Triton, and GEMM and MoE kernels use CUTLASS, TRTLLM-GEN and CuTeDSL. Speculative decoding covers n-gram, suffix, EAGLE and DFlash, and prefill, decode and encode can be disaggregated.

These are lists of capabilities, not a compatibility matrix. A given quantization format may need a particular GPU generation or a particular kernel, and the README does not say which pairs work together. Before you plan a deployment around NVFP4 or MXFP4, confirm in the documentation that your model, quantization and accelerator are supported together.

Hardware beyond NVIDIA mixes native support and plugins

The README states support for NVIDIA GPUs, AMD GPUs, Intel GPUs, and x86, ARM and PowerPC CPUs. It then adds that diverse hardware plugins exist, naming Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon and MetaX GPU.

The consequence is that non-mainstream hardware is not a single install. A plugin may live in its own repository, with its own release schedule and its own list of supported models. If you target Ascend, Gaudi, TPU or Apple Silicon, read that plugin's documentation first and expect version pairings between it and vLLM itself.

The OpenAI-compatible server, tool calling and structured outputs share one endpoint

On the usability side, vLLM offers an OpenAI-compatible API server, plus Anthropic Messages API and gRPC support. It supports streaming outputs, tool calling and reasoning parsers, and generation of structured outputs using xgrammar or guidance. Decoding options include parallel sampling and beam search. Multi-LoRA is supported efficiently for dense and mixture-of-experts layers, and distributed inference can use tensor, pipeline, data, expert and context parallelism.

The practical effect is that OpenAI-client code can be pointed at a vLLM server by changing the base URL. Compatibility is described but not itemised, so test the specific features your application depends on, such as tool calling with your model's chat template. The repository's examples directory has chat templates for several models, including chatml, falcon, chatglm and alpaca styles, and folders for tool calling, reasoning, speech to text, pooling, reinforcement learning, scale-out, Ray serving, observability and disaggregated serving. Several of these folders map to features named earlier, such as the ray_serving and disaggregated folders. A recipes directory, a benchmarks directory and a docs folder complete the layout, so you can find a worked example for most deployment shapes before writing your own.

Install with uv and download kernels, and note the build-time torch pin

The README recommends uv, with pip as an alternative, followed by a command that downloads precompiled kernels:

uv pip install vllm
vllm download-kernels  # precompiled kernels for faster startup

Building from source is also documented, and the package requires Python 3.11 or newer but below 3.15. The pyproject build requirements pin torch to exactly 2.14.0 and ask for cmake 3.26.1 or newer, ninja, setuptools-rust and setuptools-scm. A Rust toolchain file and a rust directory sit in the repository, and setup.py handles a precompiled Rust frontend.

So a source build is a heavy step: a C++ and CUDA toolchain, a Rust toolchain and one specific torch. Unless you are developing vLLM itself, take the wheel and the downloaded kernels.

Releases come every two weeks, and the newest tag is a release candidate

The recent releases are v0.33.0rc3, tagged on 2026-11-02, then v0.32.0 on 2026-10-18 and v0.31.0 on 2026-10-05. The last push was on 2026-11-02 and the project is not archived. The pace is a new minor version about every two weeks, and the 0.x series means the interface can still change between them.

The release named v0.33.0rc3 is a candidate, not a final version, so a deployment that tracks the latest tag may land on a candidate. A candidate exists to be tested by users before the final version is cut, which is a useful role if you want early access and a poor one if you need stability. Pin to a final version such as 0.32.0 for production, and read the notes when you move up. The README describes a community of many dozens of academic institutions and companies and over 2000 contributors, which gives a sense of the rate of change in the repository.

Models: 200+ architectures, with the full list kept in the docs

vLLM says it supports more than 200 model architectures from Hugging Face. The categories are decoder-only LLMs such as Llama, Qwen and Gemma, mixture-of-experts models such as Mixtral, DeepSeek-V3, Qwen-MoE and GPT-OSS, hybrid attention and state-space models such as Mamba and Qwen3.5, multi-modal models such as LLaVA, Qwen-VL and Pixtral, embedding and retrieval models such as E5-Mistral, GTE and ColBERT, and reward and classification models such as Qwen-Math.

The complete list is on the supported models page, and a new architecture may need a release that adds it. If your model is brand new, check that page and the open issues before you assume it loads. Questions and feature requests go to GitHub Issues, discussion with other users goes to the user forum, development coordination goes to Slack, and security disclosures go through GitHub Security Advisories.

Editorial conclusion

vLLM is the pick when you need to serve open-weight models to many concurrent clients behind an OpenAI-compatible endpoint, on NVIDIA, AMD or another supported accelerator. It is a heavier choice for single-user local use, and its non-NVIDIA hardware support is described partly as plugins that you must verify per device. Before adopting it, check Python 3.11 to 3.14, test your exact model and quantization on your exact GPU, and pin a stable release instead of the release candidate.

Frequently asked questions

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes