DistillKit: Offline Logit Distillation With a Compression Layer
An Open Source Toolkit For LLM Distillation
At a glance
- What is it?
- Arcee's Apache-2.0 toolkit targets the storage problem in offline knowledge distillation, packing teacher logits to roughly 300 bytes per token. The design is opinionated, and the configuration is where the real work lives.
- Who is it for?
- DistillKit fits teams that already have teacher logits or plan to capture them, and who need to train several students from one teacher without paying the VRAM cost of running both models at once. It does not fit anyone who wants a one-command distillation script, or who cannot match the compressor settings to the capture settings, since the README states that mixing configurations will not work.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 126 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The Storage Bill That Offline Distillation Sends You
Online distillation is the easy case. The teacher model sits next to the student, produces a probability distribution over the vocabulary for every position, and the student trains against it. No dataset, no disk, no compression. The cost is VRAM, and for a teacher several times the size of the student, that cost is often the whole budget.
Offline distillation removes the teacher from the training loop by capturing its outputs ahead of time. The problem the README names directly: storing top-k token and logit pairs becomes prohibitively expensive when you are distilling on billions of tokens. A single token position with a 151,936-token vocabulary, if you store a dense float32 distribution, is over 600 KB. Multiply that by a long sequence and by a corpus, and the arithmetic stops being friendly.
DistillKit is aimed at that second case. The README describes the compression system as the result of months of experimentation balancing storage cost, memory throughput, and distillation quality. The audience is teams running distillation at a scale where the capture dataset, not the GPU, is the binding constraint.
What the Compressor Actually Does to a Logit Vector
The pipeline is stated as five steps: select top-k logits, sort by log-probability with optional delta encoding, fit a polynomial to the distribution curve, quantize residuals with optional error diffusion, then bitpack into byte vectors.
The mechanism is a curve fit plus a correction. The top-k logits are sorted, and a polynomial is fitted to the shape of the distribution. The polynomial coefficients are stored at term_dtype, and the difference between the polynomial and the true value becomes a residual. Residuals are quantized, optionally with error diffusion so that quantization error is pushed into neighbouring values rather than discarded, and the whole thing is packed at arbitrary bit widths between 1 and 64 bits.
The exact_k key is the escape hatch. The top exact_k logits are stored losslessly at exact_dtype, while the remaining k minus exact_k positions are approximated. That is a deliberate choice about where fidelity matters: the head of the distribution, which carries most of the gradient signal, is kept exact, and the tail is fitted.
The README gives two reference configurations. The recommended one uses k of 128, exact_k of 16, bfloat16 exact_dtype, polynomial terms [0, 1, 2, 3, 4, "sqrt"], and lands at roughly 300 bytes per token, which the README puts at 0.15 percent of the uncompressed distribution size. The budget pick drops to k of 50, exact_k of 1, and polynomial terms [0, 1, "sqrt"], at around 114 bytes per token. The README claims the budget pick is smaller and reconstructs better than storing the top 32 logprobs in bf16. That claim is the project's own, not an independent measurement.
Online and Offline Are Two Different Programs Sharing a Loss Stack
The online path runs teacher inference during student training. The README points to examples/afm_test.yml for a complete online configuration rather than reproducing one inline, which is a gap if you are trying to decide between the two modes from the README alone.
The offline path is the one with a full worked example. A YAML file declares the student model, the dataset repo, the compressor settings, the loss functions, and the training arguments, and the CLI consumes it. The teacher block in the offline case has kind: dataset, meaning the teacher is a stored artifact rather than a live model. The dataset block points at a Hugging Face repo id, with a prepacked flag indicating the sequences are already packed to sequence_length.
The loss stack is shared and composable. Distribution losses are kl, jsd, and tvd. Ranking losses include hinge. Each entry in loss_functions carries its own weight, so the example config puts cross_entropy at 0.5 and kl at 0.5, with temperature and a missing_probability_handling key set to zero. The kl entry also carries sparse_chunk_length: 1024, which is the chunking mechanism the README mentions for processing long sequences without materializing the full distribution at once.
The README also lists hidden state alignment among the available losses, though the truncated text does not show its configuration keys.
Getting a Run Started
Installation is a clone and an editable install:
git clone https://github.com/arcee-ai/distillkit.git cd distillkit pip install -e .
Capturing your own teacher outputs requires the optional extra: pip install -e ".[capture]". The README recommends most users start with the pre-captured datasets instead, which is a reasonable default given that capture is the expensive half.
Training is a single command against a YAML file: distillkit config.yaml.
The config keys that matter most are inside teacher.logprob_compressor. The d key is the vocabulary size and must match the teacher. k is how many top logits are retained. exact_k is how many of those are stored losslessly. exact_dtype controls the storage type for the exact head. polynomial_terms is a list mixing integer exponents and the string "sqrt". residual_bins, delta_encoding, and error_diffusion toggle the optional stages, and the recommended config leaves all three off or empty.
The README is explicit that the configuration used to capture the logits must be reflected in the distillation configuration, and that mixing and matching will not work. That is the single most important operational rule in the document, and it is buried in a note at the end of the compression section rather than at the top of the configuration guide.
Where the Design Bites Back
The configuration surface is large and tightly coupled. Changing k, exact_k, or polynomial_terms changes the byte layout of the captured data, so a compressor config is effectively a schema version. If you capture with one set of terms and train with another, the README says the result will not work out, and there is no described mechanism for detecting the mismatch at load time. A team that loses the original capture config loses the dataset.
Sparse distributions are lossy by construction. The README frames sparse distillation as able to reach equivalent performance to dense given sufficient training data, which is a conditional, not a guarantee. If your corpus is small, the approximation error in the tail is not averaged away.
The online path is documented by reference to a single example file. There is no inline online YAML in the README, no description of how teacher and student are placed across devices, and no guidance on the VRAM arithmetic that the rule of thumb gestures at. If online is your intended mode, the README alone is not enough to plan a run.
No releases were retrieved for this repository, so there is no versioned changelog to consult for breaking changes to the compressor format. The last push date is recent, which suggests active development, and active development on a file format is exactly the situation where you want a version pin and a copy of your capture config.
The Alternative You Are Probably Already Using
The obvious comparison is plain Hugging Face Transformers with a custom training loop, or TRL's distillation-adjacent trainers, storing teacher logits yourself. That approach is not wrong. If your teacher is small enough to run alongside the student, online distillation in a standard Trainer subclass needs no capture step, no compressor, and no format contract between two config files.
The difference is what happens at scale. A hand-rolled pipeline that stores top-k logits as bf16 tensors pays roughly the budget-pick size the README quotes, around 114 bytes per token, and the README argues its compressed format beats that on both size and reconstruction quality. Whether that holds for your data is testable, but the structural point stands: DistillKit's contribution is the compressor plus a training loop that reads the compressed format directly, including chunked processing of sparse distributions during the loss computation. A generic Trainer does not read bitpacked polynomial residuals.
If you never intend to reuse a teacher's outputs across multiple students, the compression layer is pure overhead. The README itself frames offline as best when VRAM-limited, reusing teacher signals, or training at large scale. Reuse is the load-bearing word.
Licence, Maintenance, and What to Pin
DistillKit is Apache-2.0, which permits commercial use and modification with the usual attribution and notice requirements. The pre-captured datasets on Hugging Face are separate artifacts with their own terms, and the README does not describe them, so check the model and dataset cards before building a product on captured logits. That is a factual boundary, not legal advice.
Maintenance cost here is mostly configuration discipline. The compressor settings are a contract, and the contract is not self-describing in the material provided. Keep the capture YAML next to the dataset. Pin the DistillKit version you captured with, since no releases were retrieved and the default branch is the only reference point given.
Upgrading means re-verifying that the compressor still reads your existing captures, or recapturing. Recapturing a multi-billion-token teacher is not a small task, so the upgrade path for the dataset is the expensive part, not the pip install. The package itself is small and editable-installed from source, which makes version pinning a matter of committing a git ref rather than resolving a dependency graph.
Editorial conclusion
DistillKit fits teams that already have teacher logits or plan to capture them, and who need to train several students from one teacher without paying the VRAM cost of running both models at once. It does not fit anyone who wants a one-command distillation script, or who cannot match the compressor settings to the capture settings, since the README states that mixing configurations will not work. Before committing, verify that the pre-captured dataset you intend to use was compressed with a configuration you can reproduce in your training YAML, and confirm the capture extra installs cleanly on your Python and CUDA stack.
Community notes