AngelSlim: Tencent's Compression Toolkit for Quantization, Distillation and Speculative Decoding

The problem AngelSlim targets: compression work scattered across scripts

Compressing a large model usually means stitching together separate codebases. One repository does post-training quantization, another handles knowledge distillation, a third trains a draft model for speculative decoding, and none of them share a config format. AngelSlim's stated goal is to collapse that into a single toolkit that is, in the README's phrasing, more accessible, comprehensive and efficient. The audience is not someone who wants to run a quantized checkpoint from Hugging Face. It is the engineer who has to produce that checkpoint: pick an algorithm, run calibration or training, evaluate the drop, and export something a serving stack can load. The topics list on the repository (llm-compression, quantization, speculative-decoding, fp4, eagle, diffusion, vlm, audio) shows how far the intended scope reaches beyond text-only LLMs. That breadth is the pitch and also the main thing to scrutinize, because a toolkit covering FP8, INT4, 1.25-bit, NVFP4, ternary quantization, sparse attention and diffusion models in one repository is either a well-factored framework or a collection of research branches, and the release history is the only evidence available here for which it is.

What is actually inside: quantization, distillation, speculative decoding, sparse attention

The repository groups work into several feature areas. On quantization, the README documents PTQ support for FP8-Static, W4A8-FP8, NVFP4, INT4 and FP8, with named model coverage including Hunyuan 0.5B through 7B, Qwen2.5VL at 3B, 7B, 32B and 72B, DeepSeek-R1/V3, Kimi-K2, GLM-4.6, Qwen3-VL and Qwen3-Omni. Lower-bit research is present too: a 1.25-bit kernel called STQ1_0, a 2-bit on-device model, and a ternary quantization algorithm named TEQUILA. On distillation, the project supports both full-precision HuggingFace models and quantized QAT-style models, and as of July 2026 scale-only quantization-aware distillation on Megatron-Core for Qwen3-MoE and Hy3 with TP, EP, CP and SP parallelism. Speculative decoding is a separate track: Eagle3 training and deployment for LLMs, VLMs and audio models, plus DFlare, a block-diffusion approach with layer-wise fusion, D-Cut for adaptive verification depth pruning, and SpecExit for reasoning early exit. Stem covers sparse attention for the prefill stage by selecting top-k key blocks. The pattern is that AngelSlim is less a single pipeline than a set of named algorithms, each with its own documentation page and often its own paper.

How the pieces connect: configs, scripts and exported weights

The repository layout visible in the README points to a config-driven design. There is a configs directory with per-model subdirectories, including configs/flux and configs/seed_oss, and a scripts/ptq directory for post-training quantization entry points. Feature documentation lives under docs/source/features/, split into quantization, distill, speculative_decoding and sparse_attention subtrees, with individual pages such as features/quantization/daq.md and features/qad/mcore_qad.md. The data flow implied by that structure is: choose a model config, choose an algorithm config, run a script that loads the base weights, performs calibration or training, and writes compressed weights. Two details suggest the framework does more than emit a checkpoint. First, the distillation documentation distinguishes full-precision from QAT-style quantized students, which means the training loop has to understand the quantization scheme rather than treating it as a post-hoc step. Second, the Megatron-Core QAD path is described with TP, EP, CP and SP parallelism, so the training side is built for multi-GPU sharded runs, not single-device jobs. The speculative decoding side follows a different shape: train a draft model, then deploy it alongside the target model, which is why the Eagle3 release notes mention both training and deployment. The README does not spell out the exact intermediate artifact format shared between these stages, so treat that as something to confirm in the documentation rather than assume.

Getting it running: what the repository gives you to start from

The README does not include a pip install line or a quickstart command block, which is a real gap for a toolkit positioned around usability. What it does give is a documentation site at angelslim.readthedocs.io and direct pointers into the repository: configs/flux and configs/seed_oss for quantization configs, scripts/ptq for PTQ scripts, docs/source/features/qad/mcore_qad.md for the Megatron-Core distillation path, and docs/source/features/quantization/daq.md for the DAQ algorithm. A deployment guideline file is referenced for the Hunyuan 4 preview GGUF build at docs/source/_extra/hy4_preview_gguf_guideline.md, and the speculative decoding documentation lives under a versioned path such as angelslim.readthedocs.io/zh-cn/latest/features/speculative_decoding/dflare.html. Note that several of the documentation links in the news entries point at the zh-cn locale, so the English pages may lag. The practical starting move is to open the configs directory, find the subdirectory matching your model family, and read the documentation page for the algorithm you intend to use before writing any config of your own. Because no install command is quoted in the README, budget time for dependency resolution against your existing PyTorch and inference stack, especially if you intend to use the Megatron-Core path.

Where AngelSlim will not help you

The clearest limitation is licensing. The repository's licence field reports NOASSERTION, which means the licence could not be automatically identified from the repository contents. Until that is resolved by reading the actual LICENSE file and any per-directory notices, you cannot treat AngelSlim as safe to embed in a commercial pipeline, and you should not assume it inherits the terms of any model it compresses. A second limitation is model coverage asymmetry. The README names specific families for specific algorithms: FP8-Static and SmoothQuant for Hy3, NVFP4 for Qwen3, FP8 for Hunyuan-MT-7B, W4A8-FP8 for DeepSeek-R1/V3 and Kimi-K2. If your model is not on one of those lists, you are in unsupported territory and will be adapting configs yourself. Third, several headline items are research releases with their own branches rather than mainline features: Sherry lives on a sherry branch, TEQUILA on a tequila branch, and the STQ1_0 kernel was contributed upstream to llama.cpp as a pull request. Depending on a branch means depending on code that may not track main. Finally, the aggressive end of the compression range carries quality risk that the project itself quantifies rather than hides: the Hunyuan 4 preview build is described as compressed from 1.5TB to 214 GiB with a 0.7 percent drop on SWE-bench Pro. That is a good result for that benchmark, and it is also a reminder that sub-2-bit compression is a per-model research exercise, not a default setting.

Alternatives and how they differ in approach

The most direct comparison is with llama.cpp's quantization tooling, and the relationship is cooperative rather than competitive. AngelSlim's STQ1_0 kernel was submitted to llama.cpp as PR #22836, and the Hunyuan 4 preview GGUF build is run through Prima.cpp, a separate project, on the hardware described in the news entry. The division of labour is clear: AngelSlim produces compressed weights and, in some cases, kernels; llama.cpp and Prima.cpp consume them for inference. If your goal is to quantize a model with an established recipe and run it locally, llama.cpp's own quantization path is the shorter route and does not require you to choose an algorithm. AngelSlim is the better fit when you need to go below what the standard recipes offer, or when you need training-time methods such as distillation or draft-model training that a pure inference project does not provide. A second contrast is with the training frameworks themselves. The Megatron-Core QAD support means AngelSlim sits on top of Megatron rather than replacing it, so teams already using Megatron for pretraining can reuse that infrastructure. Teams on a different training stack would be adopting Megatron as a side effect of adopting AngelSlim's distillation path, which is a much larger commitment than the quantization features alone.

Maintenance cost and the licence question

The release cadence is visible and reasonably regular: v0.2.0 in November 2025, v0.3.0 in January 2026, and v0.5.0 in June 2026, with the last repository push in September 2026. The news log shows a steady stream of algorithm additions between those releases, which cuts both ways. You get current research, and you also get a moving target where a feature you depend on may be superseded by a newer algorithm within a couple of releases. The branch-based research releases (sherry, tequila) are the sharpest version of this problem, since they are not on main and their maintenance status is not stated. The licence situation deserves a plain statement: NOASSERTION is not a licence, it is the absence of one that tooling could recognise. Reading the LICENSE file directly is the only way to know the terms, and if the terms are unclear, that is a reason to keep AngelSlim out of a shipped product until it is clarified. Nothing here is legal advice, and the repository's own documentation is the authority on what you are permitted to do with the weights it produces, which may differ from the code licence.

Editorial conclusion

Adopt AngelSlim if you are compressing a model you control and you need PTQ, distillation and speculative decoding under one roof, particularly for Hunyuan, Qwen or DeepSeek families where the repository already publishes configs and weights. Do not adopt it if you need a settled permissive licence today, or if your target is a runtime the project has no published path to. Before committing, resolve the NOASSERTION licence field, pin the exact release tag rather than tracking main, and confirm that a checkpoint exists for your model family in the configs directory.

Official sources

1. Issues
2. Project website
3. README
4. Releases
5. Tencent/AngelSlim on GitHub