What is Distillation (in AI)? — AI Glossary

Distillation (or knowledge distillation) in AI is a technique where a smaller “student” model is trained to mimic the behavior of a larger, more powerful “teacher” model — transferring knowledge and capability into a model that’s cheaper, faster, and more practical to deploy. Instead of training the student from scratch on raw data, it learns from the teacher’s outputs (or intermediate representations), inheriting the teacher’s “knowledge” efficiently. Distillation is how Google made BERT small enough for mobile phones, and how DeepSeek-R1’s efficient smaller models were created.

How Distillation Works

The core idea: a large teacher model’s “soft” output probabilities contain more information than hard labels. When a teacher says “I’m 70% confident this is a cat, 25% a dog, 5% a fox,” that distribution is more informative than just “cat.” The student learns from these rich probability distributions, not just the final answer.

• Step 1: Train or obtain a large, capable teacher model.
• Step 2: Run your training data through the teacher to collect its outputs (soft labels) — or responses to thousands of prompts for LLMs.
• Step 3: Train a smaller student model to minimize the difference between its outputs and the teacher’s, using a combination of task loss and distillation loss (KL divergence from teacher’s distributions).
• Step 4: The student learns to approximate the teacher’s behavior in a fraction of the parameters.

For LLMs specifically, distillation often involves fine-tuning a smaller open-source base model (e.g., Llama 3 8B) on thousands of high-quality prompt-response pairs generated by a larger model (GPT-4, Claude 3 Opus). This is sometimes called “response distillation” or “instruction tuning from a teacher.”

Why Distillation Matters

The motivation is economics and practicality:

• Cost reduction: A 7B parameter model costs roughly 10-20x less per inference than a 70B model. At scale, this is transformative.
• Speed: Smaller models have lower latency — critical for real-time applications like voice interfaces and autocomplete.
• Edge deployment: Many distilled models fit in the memory of laptops or smartphones, enabling offline AI.
• Specialization: A student distilled specifically for legal document analysis can outperform its general-purpose teacher on that narrow task.

Famous distillation examples: DistilBERT (66% smaller than BERT, 97% of performance), TinyLlama, Microsoft’s Phi series (models that punch far above their parameter count), and DeepSeek’s R1-Distill series. Quantization is a complementary technique that reduces the precision of model weights (rather than the number of parameters) for additional size reduction.

Distillation vs. Fine-Tuning vs. Quantization

These three techniques are often used together but do different things:

• Distillation: Trains a new, smaller model architecture using a teacher’s knowledge. Changes model size.
• Fine-tuning: Updates an existing model’s weights on new data. Keeps model size the same; improves task-specific performance.
• Quantization: Reduces the numerical precision of weights (e.g., 32-bit → 4-bit). Keeps model architecture the same; reduces memory and speeds inference.

A common production approach: distill a large model to a smaller architecture, then quantize the distilled model for maximum efficiency. The result can be a model that runs on consumer hardware with performance close to the original giant. This is part of the broader AI infrastructure optimization toolkit.

Key Takeaways

• Distillation transfers knowledge from a large teacher model to a smaller, more efficient student model.
• Students learn from teacher’s soft probability outputs rather than just hard labels.
• For LLMs, distillation often means fine-tuning a small model on outputs generated by a large model.
• Key benefits: lower inference cost, faster latency, edge deployment capability.
• Distillation is complementary to quantization — both reduce deployment cost but through different mechanisms.

Frequently Asked Questions

Can a distilled model ever outperform its teacher?

On specific narrow tasks, yes. A student distilled specifically for medical diagnosis may outperform a general-purpose teacher on medical queries because it focuses all its capacity on one domain. For general capability, the student typically approaches but doesn’t exceed the teacher.

Is it legal to distill from commercial LLMs?

This is a contested area. OpenAI and Anthropic’s terms of service prohibit using their models’ outputs to train competing models. However, using outputs to fine-tune models for your own internal use is a gray area many companies navigate. Always check the specific API provider’s terms of service.

What is self-distillation?

Self-distillation is a technique where a model trains on its own outputs — using earlier or ensemble predictions as soft labels. It can improve calibration and performance without needing a separate larger teacher model. Useful when no larger model is available.

How does distillation relate to Microsoft’s Phi models?

Microsoft’s Phi series (Phi-1, Phi-2, Phi-3) uses high-quality synthetic training data generated by stronger models — a form of distillation — to achieve remarkable performance from very small model sizes (1.3B-14B parameters). Phi-3 Mini (3.8B parameters) matches or beats models 5-10x its size on benchmarks.

Does distillation always require the teacher’s internal weights?

No. “Black-box” distillation only requires the teacher’s outputs, not access to its internals. You generate a dataset of prompt-response pairs from the teacher API and fine-tune the student on those. This is how most LLM distillation works in practice, since frontier model weights are not public.

---

Want to go deeper? Browse more terms in the AI Glossary or subscribe to our newsletter for daily AI concepts explained in plain English.

Free download: Get the Beginners in AI Report — free daily analysis covering the latest in model efficiency and AI research.

Sources

• Grokipedia
• Stanford HAI AI Index Report

You May Also Like

• What Is Artificial Intelligence
• AI Glossary
• AI for Dummies
• How Does AI Work
• Essential AI Skills

---

Get free AI tips daily → Subscribe to Beginners in AI

Sources

This article draws on official documentation, product pages, and industry reporting. Specific sources are linked inline throughout the text.

Last reviewed: April 2026