LightCompress: A Compression Toolkit for LLMs, VLMs and Video Generators
[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.
At a glance
- What is it?
- LightCompress (formerly LLMC) collects quantization and token-reduction algorithms behind one configuration surface, then exports real quantized weights for vLLM, SGLang, AutoAWQ and MLC-LLM. Its value is breadth across model families, and its cost is that you are adopting a research toolkit, not a finished pipeline.
- Who is it for?
- Adopt LightCompress if you already have a specific model and a specific backend in mind, and you need one tool that covers integer and floating-point quantization, AWQ, GPTQ, SmoothQuant and Quarot, plus token reduction for VLMs. Do not adopt it as a general-purpose model optimizer for a model family it has never been exercised on; the release notes list DeepSeek, Mixtral, InternLM2, Qwen2VL, Llama3.2 and Wan2.1, and nothing outside that list is evidenced.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 125 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap LightCompress fills between research code and a deployable checkpoint
Quantization papers usually ship as a script attached to one model and one algorithm. Reproducing a second algorithm means reconciling a second set of dependencies, a second calibration routine and a second export format. LightCompress positions itself as the layer that absorbs that work: the README describes it as an off-the-shelf tool for compressing AIGC models, and the news entries enumerate the algorithms it has folded in, including AWQ, GPTQ, SmoothQuant and Quarot, plus integer and floating-point quantization and static per-tensor activation quantization. The audience is engineers who have a checkpoint they must shrink and an inference server they must feed, and who would rather not maintain four forks of four papers.
The scope has widened over time. The repository was renamed from LLMC to LightCompress, and the model list now spans text LLMs, vision-language models and video generative models. That breadth is the product. A team quantizing DeepSeek-V3 and a team quantizing Wan2.1 are solving different numerical problems, but they can share one config format and one export step.
What actually happens between a dense checkpoint and an exported quantized model
The pipeline implied by the documentation has four stages. You start from a dense checkpoint, load it through a configuration that names the model, the algorithm and the quantization scheme, run calibration or weight transformation over that checkpoint, and then export. The export step is where LightCompress differentiates itself from a training-time quantizer. The release notes describe exporting real quantized INT4, INT8 and FP8 (E4M3, E5M2) weights, and they name save_lightllm as a mode used to produce the published Llama-3.1-405B INT4 and INT8 checkpoints.
Two details in the news entries matter more than the algorithm list. First, FP8 weights can be loaded directly without an extra conversion step, which removes a preprocessing stage that otherwise sits between the quantizer and the server. Second, the project states that AWQ and RTN quantization of a 671B MoE model can run on a single 80GB GPU. If that holds for your checkpoint, it changes the hardware you need for the compression job itself, which is often a larger practical obstacle than the compression ratio.
For VLMs the mechanism is different. Token reduction and token pruning operate on the sequence the vision encoder or the language model sees, cutting the number of tokens rather than the precision of the weights. The README describes over 20 algorithms covering both token reduction and quantization, presented as plug-and-play strategies. Token pruning and weight quantization compose, but they fail differently: pruning degrades tasks that depend on fine spatial detail, while quantization degrades numerical stability in attention and normalization layers.
Getting a first run: Docker image, Python version, and the export target
The README gives the Docker pull commands directly. From Docker Hub: docker pull llmcompression/llmc:pure-latest. From Alibaba Cloud, which the README recommends for users in mainland China: docker pull registry.cn-hangzhou.aliyuncs.com/yongyang/llmcompression:pure-latest. The tag shown is pure-latest; the Alibaba Cloud line uses a [tag] placeholder, so you pick the version yourself.
The README recommends Python 3.11 for local development and installation, noting that it matches the project's Docker images and CI configuration and is more stable than Python 3.12 for the current dependency set. That is a specific constraint worth respecting; a 3.12 environment is a plausible source of import failures that have nothing to do with compression.
The configuration keys you will touch are only partially visible in the supplied material. save_lightllm appears as an export mode. Beyond that, the README points to the documentation site rather than listing keys inline, so the exact field names for algorithm selection, calibration dataset and quantization bit width are not something I can state from this material. Treat the docs at llmc-en.readthedocs.io as the source for those, and expect the config to be the part you spend the most time on. The backend pages for vLLM, SGLang, AutoAWQ and MLC-LLM are separate documents, which suggests the export format is chosen per backend rather than being a single universal artifact.
Where the toolkit stops being the right answer
The most concrete limitation is coverage. Every capability claim in the README is attached to a named model or family: DeepSeek-V2, V2.5, V3, R1 and R1-zero; Mixtral and other MoE models; Qwen2VL; Llama3.2; Llama-3.1-405B; InternLM2; Wan2.1. If your checkpoint is not on that list, you are extrapolating. The algorithms are general in principle, but the calibration code, the layer-name matching and the export path are usually where a new architecture breaks, and none of that is evidenced for unlisted models.
A second limit is that compression is not free of evaluation work. The README mentions support for opencompass to evaluate a compressed model, which tells you the project expects you to measure the damage rather than assume it away. A quantized checkpoint that loads successfully can still be worse at the specific task you care about, and the toolkit cannot tell you that.
A third issue is documentation depth. The README is a news feed plus a Docker command. Configuration details, calibration guidance and per-algorithm trade-offs live on a separate ReadTheDocs site, in English and Chinese. If that site lags the code, the README will not save you, because it deliberately does not duplicate the config reference. Budget time for reading two documentation sources and reconciling them.
How LightCompress differs from a single-algorithm quantizer such as AutoAWQ
AutoAWQ is the comparison the README itself invites, since it appears both as a supported export backend and as an independent project. The difference is scope of responsibility. AutoAWQ implements AWQ and produces AWQ-format checkpoints. If AWQ is the algorithm you want and the AWQ format is what your server loads, AutoAWQ does the whole job with less surface area.
LightCompress treats AWQ as one option among several and adds the surrounding machinery: GPTQ, SmoothQuant, Quarot, RTN, integer and floating-point schemes, static per-tensor activation quantization, and token reduction for multimodal models. It also emits into other projects' formats, AutoAWQ among them. That makes it a router rather than a competitor. The trade-off is real: a router has more configuration paths, more dependency versions to reconcile, and more places where a mismatch between your chosen algorithm and your chosen backend goes unnoticed until inference. If you have already decided on AWQ plus vLLM, the narrower tool is the lower-risk choice. If you are still comparing algorithms, or you need one workflow that covers both a text model and a video model, the router earns its complexity.
Maintenance, releases and what the Apache-2.0 licence leaves you to decide
The release history shows a project that ships in discrete steps rather than continuously: v1.3.0 in October 2024, v1.4.0 in February 2025, v1.5.0 in November 2025, under the older LLMC name for the first two. The last push recorded is May 2026. Between releases, the news entries show a steady cadence of new model support, which means the practical upgrade cost is not the version bump itself but re-validating your compressed checkpoint against a toolkit whose model-handling code has moved.
Pinning matters here. The Docker image is the intended distribution channel, and the README recommends Python 3.11 to match the images and CI. If you install locally instead, you own the dependency set that the image would otherwise fix for you. For a compression job you run occasionally, the image is the cheaper path.
The licence is Apache-2.0, which permits commercial use and modification and includes a patent grant. That is a permissive starting point, but it governs the toolkit, not the weights you produce from it, and not the licences of the models you compress. Check the terms attached to your base checkpoint separately. I am not giving legal advice; read the licence text and your model's terms before shipping anything.
Who should pick this up, and what to confirm before you do
LightCompress is aimed at engineers who already know which model they are compressing and which server will run it. The README's own examples are the honest test: if you are working with a DeepSeek MoE checkpoint, a Qwen2VL or Llama3.2 VLM, or a Wan2.1 video model, there is a documented path and a news entry describing it. If you are working with something else, you are the first person on that path.
The first thing to verify is the export contract. Pick your backend, read its page in the documentation, and confirm that the format LightCompress writes is the format that backend reads, before you spend GPU hours on calibration. The second is hardware: the claim that AWQ and RTN on a 671B MoE model fit in a single 80GB GPU is the single most consequential number in the README, and it is the one to reproduce on your own machine early, because if it does not hold your whole plan changes. The third is the evaluation loop. Wire up opencompass or your own harness against the compressed checkpoint first, so that when quality drops you can tell whether it was the algorithm, the calibration set, or the export.
Editorial conclusion
Adopt LightCompress if you already have a specific model and a specific backend in mind, and you need one tool that covers integer and floating-point quantization, AWQ, GPTQ, SmoothQuant and Quarot, plus token reduction for VLMs. Do not adopt it as a general-purpose model optimizer for a model family it has never been exercised on; the release notes list DeepSeek, Mixtral, InternLM2, Qwen2VL, Llama3.2 and Wan2.1, and nothing outside that list is evidenced. Before committing, verify three things against your own checkpoint: that the export mode you need (save_lightllm, or the INT4/INT8/FP8 paths documented for vLLM and SGLang) produces weights your inference stack actually loads, that the calibration set you feed the quantizer matches your deployment distribution, and that the algorithm you pick is one the docs describe for your architecture rather than one you inferred from the topic list.
Community notes