vLLM review: PagedAttention, continuous batching, and the cost of flexibility

What vLLM actually solves

vLLM addresses a specific bottleneck: the memory cost of attention key and value tensors during LLM serving. In a typical inference server, KV cache allocation is static and wasteful. Requests arrive, get a fixed chunk of memory, and that memory stays reserved even if the request finishes early. vLLM's PagedAttention, described in the project's paper, manages KV memory in fixed-size blocks, similar to how an operating system pages virtual memory. This allows memory to be allocated and freed as requests progress. The result is higher throughput because more requests can fit in the same GPU memory. The project targets teams running production inference services, not individual users experimenting with a single model. It is for people who need to serve many concurrent requests, support multiple models, or run distributed inference across several GPUs.

How PagedAttention and continuous batching work

The core mechanism is PagedAttention. Instead of allocating contiguous memory for each request's KV cache, vLLM divides memory into blocks and maps them non-contiguously. This eliminates fragmentation and allows sharing of blocks across requests, for example during parallel sampling where multiple outputs share the same prefix. The README lists continuous batching, chunked prefill, and prefix caching as features built on top of this. Continuous batching means the engine can add new requests to the batch as others finish, rather than waiting for a fixed batch to complete. Chunked prefill splits long prompts into smaller pieces so they do not block decode of other requests. Prefix caching reuses the KV cache for repeated prompt prefixes. These mechanisms work together to keep GPU utilization high. The README also mentions piecewise and full CUDA/HIP graphs, which reduce kernel launch overhead. This is not a simple queue-based server; it is a scheduling system with memory management at its core.

Getting it running: install and first serve

Installation is straightforward if you have a Python environment. The README recommends using uv, a Python package manager, with the command `uv pip install vllm`. Alternatively, you can use pip with `pip install vllm`. For development, you can build from source, and the documentation has a specific page for building a wheel from source on GPU. After installation, the typical workflow is to start an OpenAI-compatible API server. The README does not give the exact serve command, but the documentation quickstart does. Based on the repository layout, the common pattern is `vllm serve <model_name>`, where model_name is a Hugging Face model identifier. The server then exposes endpoints compatible with the OpenAI API, including streaming and structured outputs via xgrammar or guidance. You can also use the engine programmatically through the Python API. The README emphasizes integration with Hugging Face models, so you can point it at any supported architecture. There is no need to convert weights to a custom format, which is a major convenience.

Parallelism and hardware: the breadth and the caveats

vLLM supports multiple forms of parallelism: tensor, pipeline, data, expert, and context. This means you can split a single model across GPUs (tensor and pipeline), replicate the model for more throughput (data), or distribute experts in a MoE model (expert). Context parallelism likely handles very long sequences. The README also lists support for NVIDIA, AMD, Intel GPUs, and x86/ARM/PowerPC CPUs, plus plugins for Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, and others. This breadth is impressive, but it comes with a caveat: not all features work on all hardware. For example, the optimized attention kernels like FlashAttention and FlashInfer are likely tuned for NVIDIA. The README does not specify which kernels are available on which hardware. If you are on an AMD GPU, you may get a different kernel or a slower fallback. Similarly, quantization methods like FP8 and INT8 may not be supported on every hardware plugin. The documentation is the place to check, but the README itself is silent on these constraints. This is a trade-off: broad hardware support is a strength, but it also means the feature matrix is complex and you need to verify your specific combination.

Model coverage and quantization: what you can run

The README claims support for over 200 model architectures on Hugging Face. This includes decoder-only models like Llama and Qwen, MoE models like Mixtral and DeepSeek-V3, hybrid attention and state-space models like Mamba, multi-modal models like LLaVA and Pixtral, and even embedding and reward models. That is a wide net. The project maintains a full list in the documentation. Quantization support is also extensive: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, and TorchAO. This means you can run quantized models for lower memory footprint and potentially higher throughput. However, the README does not state that all quantization methods work with all model architectures. For instance, GGUF is often used with llama.cpp-style models, and GPTQ/AWQ require calibration. The practical implication is that you need to check both the model and the quantization method before assuming it works. The project's speed of development, with releases v0.28.0 and v0.27.x in the same month, suggests that support is actively expanding, but also that the feature matrix changes frequently.

Limitations and failure modes

The most obvious limitation is that vLLM is not a fit for every model. If your model is not in the supported list, you cannot use vLLM without writing custom code. The README lists many architectures, but it does not cover every model on Hugging Face. For a custom or niche model, you would need to either wait for support or use a different engine. Another limitation is the complexity of the serving stack. The README lists many features, but each one adds configuration options and potential failure points. For example, disaggregated prefill and decode separates the prefill and decode phases into different processes or machines. This can improve throughput for long prompts, but it introduces network communication and coordination overhead. If you do not need that scale, the extra complexity is a burden. Also, the README mentions speculative decoding with EAGLE and other methods. These require additional models and careful tuning to get speedups; they can actually slow down inference if not configured correctly. Finally, the project's fast release cadence means that upgrading can bring breaking changes. The documentation likely has migration notes, but the README does not mention them.

Alternatives and how they differ

The primary alternative is Hugging Face's Text Generation Inference (TGI), which also serves LLMs with an OpenAI-compatible API. TGI uses its own continuous batching and memory management, but it does not use PagedAttention. Instead, TGI uses a different approach to KV cache management, often relying on the model's attention implementation. This means TGI may have lower memory efficiency for very long contexts or high concurrency, but it is often simpler to deploy because it is a single Docker container with fewer configuration options. Another alternative is llama.cpp's server, which is designed for CPU and Apple Silicon, and uses a different memory layout (mmap) and batch scheduling. llama.cpp is much lighter and easier to compile, but it does not support the same breadth of models or parallelism options. The key difference is that vLLM is built for scale: distributed inference, multiple GPUs, and high request rates. TGI and llama.cpp are better for smaller deployments or single-node serving. If you need multi-GPU tensor parallelism or MoE expert parallelism, vLLM is more mature in that area.

Maintenance and license considerations

vLLM is licensed under Apache-2.0, which is permissive for commercial use, modification, and distribution. The license does not require you to open-source your own code, but it does require preserving copyright notices and stating changes if you redistribute the project. This is a low-license-friction option for companies. The project is actively maintained, with a release every few weeks. The last push was August 2026, and there are three releases in the same month. That is a high maintenance cadence. For adopters, this means you need to track releases and test upgrades. The README does not provide a migration guide, but the documentation likely does. The project also has a large contributor base of over 2000 contributors, which suggests that maintenance is not a single-vendor risk. However, the complexity of the codebase, with custom CUDA kernels and graph transformations, means that debugging issues may require deep expertise. The project offers a user forum and Slack for support, which is helpful, but there is no commercial support contract listed.

Editorial conclusion

Adopt vLLM if you need high-throughput serving of popular Hugging Face models on NVIDIA, AMD, or Intel GPUs, or on CPU, and you can invest in learning its configuration and monitoring. Do not adopt it if you need a simple single-model server with minimal setup, or if your models are not in the supported list. Before adopting, verify that your exact model architecture and hardware are listed in the supported models documentation, and check the latest release notes for breaking changes in the OpenAI-compatible API or parallelism options. The engine's throughput advantages come with real complexity, so budget time for tuning and testing against your workload.

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes