llm-compressor: Quantizing Hugging Face Models for vLLM Deployment

The gap between a full precision checkpoint and a servable vLLM model

A Transformers checkpoint on the Hugging Face Hub is stored at whatever precision the publisher chose, usually bfloat16 or float16. vLLM can serve that directly, but the memory footprint and the per token cost are set by the stored precision. Getting to int8, FP8, NVFP4 or W4A16 is not a matter of flipping a flag at load time. The weights have to be transformed, and for most schemes a calibration pass has to run first so the quantizer knows the activation ranges it is working with.

llm-compressor exists to perform that transformation and to write the result in a format vLLM understands. The README describes it as the library for optimizing models for deployment with vLLM, covering weight, activation, KV cache and attention quantization, with models saved in the compressed-tensors format. The audience is therefore narrow and specific: engineers who have a model they intend to serve on vLLM and who need it smaller or faster, and who are willing to run a calibration job to get there. If you are serving on a different runtime, the output format is the wrong output format, and this is not your tool.

Recipes, modifiers and the calibration pass

The unit of work is a recipe. A recipe names the algorithms to apply, and the library applies them as modifiers during a forward pass over calibration data. The README points at one modifier in particular as a readable example of the pattern: REAP expert pruning lives at src/llmcompressor/modifiers/pruning/reap, and the README says it can be used as a template for implementing other expert pruning algorithms. That tells you something about the architecture. Modifiers are pluggable, they hook into the model during calibration, and the repository expects people to write their own when the shipped set does not cover a case.

REAP itself is worth describing precisely, because it is the one algorithm the README explains in mechanism rather than in name. It reduces VRAM for Mixture-of-Experts models by structurally removing less relevant experts in each layer. Relevancy is proxied by a saliency metric computed from calibration forward pass data. The user sets a target expert sparsity, and the algorithm removes experts to hit it while trying to limit the damage from the ones it removes. The README gives a worked combination: Qwen3.8-2.4T-A95B-NVFP4-REAP-25 prunes 25 percent of the least salient experts before quantization, which reduces VRAM further than quantization alone while, in the README's phrasing, maintaining accuracy recovery.

The data flow that matters for planning is this. You supply a model, a dataset for calibration, and a recipe. The library runs the calibration pass, the modifiers compute their statistics or make their structural changes, the quantized weights are produced, and the result is saved in compressed-tensors format. Everything downstream is vLLM's problem. The cost of that flow is the calibration pass itself, which is a full forward pass over your calibration set, and for a large model that is the dominant expense.

Running a compression job: the shape of the commands

The README does not reproduce a full command line, but it does point at runnable example scripts, and those are the honest starting point. The examples directory is organized by precision and by problem: examples/quantization_w4a16/qwen3_8_gptq_awq_example.py for INT4 weight-only, examples/quantization_w8a8_fp8/nemotron_3_5_lightning_example.py for FP8, examples/quantization_w4a4_fp4/hy3_example.py for the NVFP4 plus FP8 combination, examples/quantizing_moe/glm5_example.py and examples/quantizing_moe/qwen_3_8_example.py for Mixture-of-Experts models, examples/reap_expert_pruning/qwen38_example.py for pruning combined with quantization, and examples/model_free_ptq/muse_glimmer_fp8_block.py for the model free path.

That directory layout is the real documentation of how you get it running. You pick the example whose precision and architecture family match your target, you change the model identifier, you change the calibration dataset, and you run it. The library is installed from PyPI as llmcompressor, and the README links its documentation at docs.vllm.ai/projects/llm-compressor. Because the examples are the interface, the practical advice is to read the example before reading the API reference: the example encodes which recipe goes with which precision, and that mapping is not something you want to guess at.

Two operational features are named in the README and change how you plan the job. The first is DDP support, which lets the calibration pass run across multiple GPUs. The second is disk offloading, which lets weights live on disk rather than in VRAM during compression. The README cites GLM-5.2 as the case where these matter: the full precision model requires 1.6T of VRAM, and the checkpoint was produced using DDP plus disk offloading in under 2 hours, with NVFP4 quantization of MoE layers and FP8 quantization of attention layers cutting model size by more than 70 percent. Treat that as a claim from the project about one model on one cluster, not as a general throughput figure. It does establish that the combination is the intended answer when the model does not fit.

What the supported precision list does and does not tell you

The README's supported precisions section begins with activation quantization at W8A8 in both int8 and fp8 variants. The surrounding text names NVFP4, FP8, INT4, W4A16 and W4A4 FP4 across the various checkpoint announcements. The important reading is that the list is a boundary, not a menu. A precision that does not appear in the README is a precision you should assume is unsupported until an example shows otherwise.

The second boundary is hardware. NVFP4 and FP8 are not universally available on every accelerator, and the README does not provide a compatibility matrix tying each precision to a GPU generation. That is a genuine gap. You can read the checkpoint announcements and see that NVFP4 is used for MoE layers and FP8 for attention on several large models, which suggests the pairing is a common pattern, but the material does not state which hardware those checkpoints were validated on. Verify that yourself before you plan a deployment around NVFP4.

The third boundary is architecture. The examples cover a specific set of model families, and the README mentions that some of these are multimodal, such as Muse-Glimmer-30B being enabled for single-GPU deployment by FP8, NVFP4 and INT4 checkpoints. Multimodal support is therefore present in at least some cases, but the README does not describe how calibration data is handled for the vision tower versus the language tower. If your model is multimodal and not one of the named ones, expect to spend time on the recipe.

Where llm-compressor is the wrong tool

The clearest failure mode is format lock-in. Output is compressed-tensors, and the README frames compatibility in terms of vLLM. If your serving stack is TensorRT-LLM, ONNX Runtime, or a bespoke inference engine, the artifact this library produces is not the artifact you need, and no amount of recipe tuning changes that. You would be better served by a quantization path native to your runtime.

The second case is a model the examples do not cover. The modifier system is designed to be extended, and the README explicitly presents the REAP implementation as a template for writing your own pruning algorithm. That is an invitation, but it is also a statement about coverage: if your architecture needs a modifier that does not exist, you are writing it. Budget for that.

The third case is accuracy sensitivity without an evaluation plan. The README repeatedly uses the phrase accuracy recovery when describing quantized checkpoints, and cites GPQA for GLM-5.2. Accuracy recovery is not the same as no accuracy loss. A calibration set that does not resemble your production traffic will produce activation statistics that do not resemble your production activations, and the resulting model can look fine on a benchmark and behave differently on your inputs. The library gives you the mechanism; it does not give you the eval. If you cannot measure the delta on your own task, you are quantizing blind.

A fourth, milder case: if the model already fits comfortably and latency is acceptable, compression adds a calibration job, a new artifact to version, and a new failure surface, in exchange for savings you do not need.

The alternative: quantize at load time in the serving engine

The realistic alternative for many teams is to skip ahead-of-time compression and let the serving engine quantize on the fly. vLLM itself supports several quantization modes that are configured at load time, where the engine reads the checkpoint and applies the scheme as the model is instantiated. The difference in approach is fundamental. Ahead-of-time compression, which is what llm-compressor does, runs a calibration pass over real data, computes activation statistics, and bakes the result into a standalone artifact that any vLLM instance can load without repeating the work. Load-time quantization does no calibration pass; it applies a scheme with fixed or heuristically derived parameters as the model comes up.

That difference decides the trade-off. Load-time quantization costs nothing to try and requires no artifact management, but without calibration it generally cannot reach the same accuracy at aggressive bit widths, and it cannot do structural changes at all. REAP expert pruning is the sharpest example: removing experts from a Mixture-of-Experts model is not something a loader can do, because it changes the model's structure rather than the numeric type of its weights. If your goal is FP8 or W8A8 and your engine supports it at load time, try that first; it is a smaller commitment. If your goal is W4A16, NVFP4, or expert pruning, ahead-of-time compression is the only route, and llm-compressor is the route that produces vLLM-compatible artifacts.

Release cadence, licence and what to check before you start

The repository is on the main branch, not archived, with a last push in September 2026. Recent releases are 0.13.0 in August 2026, 0.12.0.1 in late July 2026, and 0.10.0.3 in late July 2026. The 0.10.0.3 to 0.12.0.1 to 0.13.0 sequence within roughly two weeks indicates active development, and it also indicates that pinning a version matters. A library that moves this fast can change recipe behaviour between minor releases, and a recipe that produced an artifact last month may produce a slightly different one today. Pin llmcompressor in your environment and record the version alongside any checkpoint you publish.

The licence is Apache-2.0. That is a permissive licence, and it is the same family as much of the surrounding tooling. The practical implication is that you can use the library and redistribute compressed checkpoints without a copyleft obligation on your own code. This is a description of the licence identifier, not legal advice; if you are redistributing quantized weights derived from a model with its own licence terms, the model licence is a separate question from the library licence, and the library's licence does not answer it.

The upgrade cost is concentrated in the recipes. Because the API is expressed through example scripts and modifier classes, a minor version bump can change which recipe keys are valid or how a modifier is configured. The mitigation is to keep your recipe in version control next to the example it was derived from, so that a diff against the upstream example shows you what changed. Before adopting, the three things to verify are concrete: confirm an example exists for your architecture family, confirm the precision you want is both listed in the README and supported by your hardware, and confirm that the compressed-tensors output from your chosen recipe loads in the vLLM build you actually run.

Editorial conclusion

Adopt llm-compressor if you already serve models with vLLM and need weights, activations, KV cache or attention quantized, and your architecture appears in the examples directory. Do not adopt it if you need a runtime that is not vLLM, or if your target precision is not among the ones the README lists. Before committing, verify three things: that your model class is covered by an existing example, that your GPU generation supports the precision you picked, and that the compressed-tensors output loads in your vLLM build. Start from the closest example script rather than writing a recipe from scratch.

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. vllm-project/llm-compressor on GitHub