LLM Compressor: quantizing models for vLLM inference
Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM
At a glance
- What is it?
- LLM Compressor provides algorithms and workflows for compressing language models, including weight and activation quantization, with output formats designed for use with vLLM.
- Who is it for?
- LLM Compressor can help produce quantized checkpoints for supported vLLM deployments. Select a scheme compatible with the model and target hardware, then evaluate task quality, memory, latency and throughput against an uncompressed baseline.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 14, 2026, and from our analysis. They are not legal advice.
Star history
- vllm-project/llm-compressor3,966 ★ · +188
| Repository | 2026-09-13 | 2026-11-14 |
|---|---|---|
| vllm-project/llm-compressor | 3.8k | 4k |
188 Stars in the last 62 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
What LLM Compressor does
LLM Compressor is a library for compressing models for inference, with a focus on quantization and integration with the Hugging Face and vLLM ecosystems. The project describes techniques for weights, activations, key-value caches and attention, and provides examples for different model families and hardware.
Quantization represents values at lower precision to reduce memory or improve inference efficiency. The practical gains and any quality loss depend on the scheme, model, workload and supported kernels.
Installation and environment
The README documents `pip install llmcompressor`. Install in an isolated environment and consult the current installation guide for compatible Python, PyTorch, accelerator and driver versions. A package install alone does not establish that a particular GPU or quantization scheme is supported.
For source builds, record the exact commit, build configuration and resulting package version. A version derived from Git can depend on build state; inspect the built artifact instead of assuming its version from a branch or tag.
Choosing a quantization scheme
The project documents weight-only and weight-and-activation schemes including INT4, FP8 and FP4 variants, with algorithms such as GPTQ and AWQ. Availability and performance vary by model architecture, serving version and hardware. Choose a documented example that matches the target, and verify the corresponding vLLM support matrix.
Different formats can trade memory, accuracy, calibration cost and throughput differently; a smaller bit width does not guarantee a faster model on every device.
Calibration and quality
Some algorithms use calibration data to choose quantization parameters. Use representative, appropriately licensed data and keep a reproducible record of samples, preprocessing and settings. Evaluate the compressed model on held-out tasks and safety criteria relevant to your application.
Observer and kernel implementation details change between releases. Consult the specific release notes and source for a setting's definition rather than interpreting an error buffer as a count of kernel failures.
Saving and serving a checkpoint
The README shows applying a quantization recipe and saving a checkpoint in compressed-tensors format, then loading compatible checkpoints in vLLM. Compatibility still depends on the model, quantization scheme, hardware and versions of both tools. Preserve the tokenizer and configuration alongside weights, and run a load and generation check in the intended serving environment.
The project also publishes example checkpoints; compare using an existing checkpoint with the cost and control of compressing your own.
Testing and reproducibility
The repository organizes tests by purpose and hardware requirements. Default make or pytest targets may omit expensive suites; inspect the target and current test configuration, then explicitly run the relevant unit, integration, example and accelerator tests. A green subset does not establish coverage of every model or device.
For reproducibility, capture the model revision, dataset, recipe, package versions, hardware and commands, and validate the resulting artifact independently.
Performance and deployment tradeoffs
Release notes may report speedups for specific kernels or internal benchmarks. Treat those numbers as workload-specific until the model, accelerator, input sizes and measurement method are known. Benchmark end-to-end inference and compression time on representative hardware, and report latency, throughput, memory and quality together.
Review the project's current documentation and Apache-2.0 license, as well as the separate terms for model weights, datasets and dependencies before deployment or redistribution.
Editorial conclusion
LLM Compressor can help produce quantized checkpoints for supported vLLM deployments. Select a scheme compatible with the model and target hardware, then evaluate task quality, memory, latency and throughput against an uncompressed baseline. Pin the toolchain and model revision, review the relevant example and license, and verify the saved artifact with the intended serving version.
Frequently asked questions
what is llm compressor
It is a library for optimising models for deployment with vLLM, presented as fast and easy to use. It provides quantisation algorithms and transforms spanning weights, activations, key-value cache and attention, integrates with Hugging Face models and repositories, supports distributed and disk-offloaded compression for very large models, and saves output in a compressed-tensors format that vLLM can consume.
how to install llm compressor
The project README documents `pip install llmcompressor`. Check the current installation guide for supported Python, accelerator, CUDA and dependency versions, and use an isolated environment.
Which quantization formats does llm-compressor produce?
The examples are organised by technique and cover weight-only four-bit cases, the NVFP4 and MXFP4 four-bit floating-point formats, eight-bit floating-point and integer formats, eight-bit mixed floating-point, key-value cache quantisation, attention and embedding quantisation, non-uniform schemes, and activation-aware and automatic-round methods. The release notes also describe grid-search scale selection for the MSE and importance-matrix observers.
How does llm-compressor integrate with vLLM?
Through the saved artefact rather than through a runtime dependency. Models are written in the compressed-tensors format, which is stated to be compatible with vLLM, and the project publishes production-ready quantised checkpoints as worked examples. It also supports Hugging Face models and repositories, so the unquantised starting point is loaded through the usual ecosystem.
How are llm-compressor's tests organised?
The repository has tests and markers for different workloads. The exact default selection and hardware requirements are version-specific; inspect the current Makefile and test configuration and run the suites relevant to your change rather than assuming a default run covers everything.
Official sources
Was this useful?