How do large models "teach" small models? The art of knowledge transfer
8 min readLarge language models can have hundreds of billions of parameters. To deploy them on phones, laptops, or embedded devices, we need to make them smaller. The industry relies on three main techniques: Quantization, Pruning, and Distillation.
Quantization reduces numerical precision — compressing FP16 weights down to INT4 or lower. The model architecture stays the same; each parameter simply takes up fewer bits. We covered this in detail in our Quantization article.
Pruning is like trimming a bonsai tree — removing neurons and connections that contribute minimally to the output, making the network leaner.
Distillation is the most knowledge-intensive approach: training an entirely new, smaller model to mimic the behavior of a large one. The large model acts as the teacher, the small model as the student, condensing a lifetime of experience into a compact form.
Same image, less data. Model structure unchanged; weights go from high to low precision.
Remove unimportant weights and neural connections to create a more compact network.
Train a small model to mimic a large model's output behavior. Transfer knowledge, not parameters.
Quantization reduces brush strokes, pruning removes chapters, distillation is the master condensing a lifetime of wisdom to teach an apprentice.
The core idea of distillation comes from Hinton et al.'s seminal 2015 paper. It defines two roles:
Teacher Model: A large, pre-trained model — say, a 70B-parameter LLM. Powerful but bulky, slow to run inference.
Student Model: A much smaller model — 7B or even 1.5B parameters. Its goal is to achieve performance as close to the teacher as possible with far fewer parameters.
Traditional training uses hard labels — the model learns only "what the right answer is." But the teacher's output is a full probability distribution. This distribution contains what Hinton called "dark knowledge" — the teacher's confidence across all options reveals relationships between classes that hard labels simply cannot convey.
The temperature parameter T controls how "soft" the distribution becomes. Higher T produces a smoother distribution with richer dark knowledge; T = 1 is the original distribution.
Teacher model predicting the next word — "The capital of France is ___":
Answer: Paris
Only tells the student "the answer is Paris."
Paris 85%, London 8%, Berlin 5%, Tokyo 2%
Student also learns: London and Berlin are also plausible capital answers.
Hard labels are the answer key to an exam. Soft labels are the teacher's explanation — why it's right, why it's wrong, and which other answers also make sense.
Depending on how the student learns from the teacher, distillation approaches fall into several categories:
The classic approach. The student matches the teacher's output probability distribution, using KL Divergence to measure the gap between the two distributions. This is the most direct way to "mimic how the teacher speaks."
A deeper form of imitation — the student learns not just the teacher's output but also its intermediate representations. This is like mimicking not just what the teacher says, but how the teacher thinks.
Common in modern LLM distillation. The student generates its own text, and the teacher scores it. This lets the student learn from its own mistakes — like doing practice problems and having the teacher grade them.
A model distills itself into a smaller version — no external teacher needed. The larger version of the model serves as teacher, training a structurally similar but smaller student variant.
The distillation training objective combines two loss components:
● First term: cross-entropy between student and ground truth — learn the correct answer
● Second term: KL divergence between student and teacher soft distributions — transfer dark knowledge
α = 0.5: hard labels and soft labels weighted equally
Logit distillation mimics how the teacher speaks. Feature distillation mimics how the teacher thinks.
Distillation has been widely adopted for production deployment of leading models:
DeepSeek's 671B reasoning model was distilled into a series of smaller models: R1-70B, R1-32B, R1-14B, R1-8B, and R1-1.5B. From 671B to 1.5B, the parameter count shrinks by 447x, yet the smaller models retain surprisingly strong reasoning ability.
Alibaba's Qwen3.5-0.8B / 2B and other small models are distilled from larger Qwen models, specifically optimized for edge deployment. They run on smartphones while significantly outperforming same-sized models trained from scratch.
Google's Gemma series is designed for on-device use, leveraging knowledge from the Gemini family via distillation to maximize capability retention while staying compact.
Key benchmark: a well-distilled 7B model typically retains 85-95% of a 70B teacher model's capability.
Distillation is like extracting the essence from a full bottle of wine — 10x smaller in volume, but 90% of the flavor is preserved.
The teacher's probability distribution contains rich dark knowledge, far more valuable than simple correct answers.
Through distillation, a 1.5B-parameter model can inherit most of the reasoning ability of a 70B model.
First distill to shrink the model, then quantize to compress weights. This is the golden combo for on-device AI.
A model's knowledge matters far more than its parameter count. Distillation proves knowledge can be efficiently compressed and transferred.
The essence of distillation: you don't need to relearn everything from scratch — you just need to stand on the shoulders of giants.