What is Knowledge distillation?
Knowledge distillation is a training technique in which a small model, the student, is trained to imitate the output behaviour of a large trained model, the teacher, so the small one gets most of the accuracy for a fraction of the serving cost. The open-source tooling splits by domain, from vision training pipelines to compression libraries aimed at inference runtimes.
How distillation works: soft targets and a second loss
Distillation starts with two models. The teacher is large, already trained and expensive to run. The student is small and cheap. The training signal for the student is not the dataset's labels alone but the teacher's output distribution for each input. Where ordinary training pushes the student towards the correct answer, distillation pushes it towards the shape of the teacher's whole output: the weight it puts on wrong answers, and how it ranks them against each other.
Two knobs make that work. Temperature rescales the teacher's logits before the softmax, so the distribution is softer and the relative weights of the wrong answers become visible; the same temperature is applied to the student. The loss is then a weighted sum of two terms, the ordinary loss against the true labels and a divergence between the student's softened distribution and the teacher's. Raise the weight on the second term and the student imitates the teacher more closely, its mistakes and biases included. Lower it and the student is mostly a supervised model that happens to have a teacher in the room.
This is why distillation is not compression in the storage sense. A quantized model is the same weights at lower precision. A distilled model is a different, smaller network trained to behave like another one, which is how it reaches compression ratios that precision tricks do not, and also how it fails in ways precision never does.
The families of method, and what each one transfers
The methods divide by what they transfer.
Response-based methods use the teacher's output distribution. That is the setup above, and it is the cheapest to run because it needs nothing beyond teacher outputs on your data.
Feature-based methods add a term that matches intermediate representations, so the student is asked to reproduce not only the final distribution but hidden states on the way there. More gets transferred, and more is demanded of the architecture, because the layers have to line up.
Relation-based methods transfer relationships between examples rather than values, asking how one input's representation compares with another's. The intuition is that what generalises is structure rather than content.
The pipelines around them are more varied than the taxonomies suggest. A project may pre-train a teacher, distil into an intermediate model of a different size, and distil again, or fold distillation together with pruning, quantization or speculative decoding in one compression pass. The classification above describes the loss, not the schedule, and most real tooling blurs the lines.
When you need it, and when you do not
Distillation earns its cost in a few specific situations.
Where the deployment budget is the constraint. If a model answers every request of a service, cutting its size or its latency by a large factor pays for a training run. That is the case the technique was built for.
Where you cannot ship the teacher. Licence, hardware or data-handling constraints sometimes mean the large model stays behind your firewall and the student is all that reaches users. Distillation is the standard answer, because the student learns from the teacher's outputs rather than from its weights.
Where the student has to be trained anyway. A task-specific model that would be expensive to build from scratch can be initialised from a teacher trained on something broader.
Where it is usually not worth it. A small dataset, because a student has fewer parameters to absorb the signal with. A task where the teacher is barely better than a well-trained small model. And a team with no evaluation harness, because the characteristic failure of distillation is silent: a student that matches its teacher and still loses to a baseline nobody compared it against.
Limits worth planning for
Four limits deserve a plan rather than a surprise.
The student inherits a ceiling. Distillation transfers a limit as well as a starting point. If the teacher's errors are systematic, the student reproduces them, and it reproduces them consistently, which is worse for a system that might have learned the pattern by another route.
The data cost is real. Producing teacher outputs for a training set is inference over the whole set with the most expensive model you own. On a large corpus that is the dominant line in the budget, and it is paid before the student learns anything.
The hyperparameters decide more than the method. The temperature, the weight on the distillation term and the number of stages are where most of the outcome is decided, and they are usually tuned on a small validation slice. Get them wrong and you get a student worse than either extreme: a model that has picked up some of the teacher's noise without learning its structure.
Evaluation is not optional. Comparing a student to its teacher measures imitation. Comparing it to a baseline trained without a teacher measures whether the exercise was worth doing. Those are different numbers, and the second is the one that decides whether you ship.
How the tooling splits, and where a distilled model ends up
The tooling divides by domain and by how much of the pipeline it owns.
yoshitomo-matsubara/torchdistill is the research-shaped one: a coding-free framework built on PyTorch that turns knowledge distillation and plain training experiments into declarative YAML configs. Its description records 26 distillation methods implemented so far from work presented at venues including TPAMI, CVPR, ICLR, ECCV, NeurIPS, ICCV and AAAI, with trained models, training logs and configurations published so that results can be reproduced. Our analysis of torchdistill puts the trade-off plainly: you adopt its configuration vocabulary and its dependency floor.
lightly-ai/lightly-train is a vision pipeline in one Python package, covering self-supervised pretraining, detection and segmentation fine-tuning, and distillation for model families including YOLO, ViTs, RT-DETR and DINOv3. Our analysis of lightly-train notes that the AGPL-3.0 licence is the first thing to check before you build a product on it.
NVIDIA/Model-Optimizer sits at the deployment end. It is described as a unified library of model optimization techniques including quantization, distillation, pruning, neural architecture search and speculative decoding, compressing models for downstream frameworks such as TensorRT-LLM, TensorRT, vLLM and SGLang. Our analysis says it turns a Hugging Face, PyTorch or ONNX checkpoint into a compressed one that those runtimes can serve, that the value is real, and that the useful path runs through NVIDIA's stack with documentation that is uneven outside the LLM examples.
sauravpanda/BrowserAI is at the far end of that chain. Our analysis describes it as an MIT-licensed TypeScript library that loads MLC, Transformers, Flare and Demucs engines in the browser and exposes them behind one API, and the models it lists include deepseek-distill, which makes it an example of a distilled checkpoint being served rather than trained.
In practice
Distillation is a way to move capability from a model you can afford to train into one you can afford to run, and the decisions that matter are the temperature, the weight on the teacher's distribution, the size of the student and the baseline you compare against. Read torchdistill if you want the methods side by side with configs you can rerun, lightly-train for the vision path, and Model-Optimizer when the goal is a checkpoint an inference runtime can serve. Before starting, cost the teacher inference over your whole dataset, and put in place the comparison against a non-distilled baseline first, because without it the result is unfalsifiable.