Start a project
Techniques Reviewed 24 Sep 2026 · 17 sources

distillation

/ˌdɪ.stəˈleɪ.ʃən/ noun Plural distillations · verb distill (also distil) · adjective distilled
Definition

A technique that trains a small AI model to reproduce a larger model’s outputs, so it approaches the larger model’s ability at lower cost.

Examples
  • DistilBERT, 40% smaller and 60% faster than BERT
  • Gemini 1.5 Flash, trained by Gemini 1.5 Pro
  • GPT-4o mini fine-tuned on outputs from GPT-4o
  • Six small Qwen and Llama models trained on DeepSeek-R1’s outputs
A large model sits at a desk and hands its answer sheet to a much smaller model beside it. The small model copies each answer onto its own sheet.
Inkling · A small model learns by copying a large model’s answersFig. D1
Earliest AI use
2014
Named in a paper for a NIPS 2014 workshop, posted to arXiv in March 2015.
Current meaning
2016
Training on text a teacher generates, not only on its probabilities.
Coined by
Hinton et al.
Geoffrey Hinton, Oriol Vinyals and Jeff Dean.
Also called
Model distillation
OpenAI’s product name. Research papers often say knowledge distillation.
§01

In plain words

In AI, distillation is a way to make a small, cheap model by training it on the outputs of a larger, more capable one.

Large AI models are costly to run. Distillation puts much of their ability into a smaller model. A large, capable model, called the teacher, produces answers for a big set of inputs. A smaller model, the student, is trained on those answers until it responds in much the same way. The student needs less memory and computing power, so it is faster and cheaper to serve, though usually somewhat less capable. DistilBERT, released by Hugging Face in 2019, is 40% smaller and 60% faster than the BERT language model it came from. It keeps 97% of BERT’s language understanding.

Leading labs use it to make cheaper versions of their own models. Google trained Gemini 1.5 Flash from Gemini 1.5 Pro. OpenAI lets developers train its cheaper models on the outputs of its larger ones. DeepSeek released six small models distilled from R1, its reasoning model.

Since a student can learn from answers alone, a company can also distill a rival’s model by querying it at scale. OpenAI’s terms of use forbid using its output to develop competing models.1 In February 2026 OpenAI sent House lawmakers a memo accusing DeepSeek of using distillation to mimic its products.2 DeepSeek’s researchers have said that R1’s success did not depend on rivals’ output.3 Anthropic has reported that three labs used about 24,000 fraudulent accounts to distill its Claude models.

Distillation is often confused with quantization, which keeps a model’s weights and stores each one in fewer bits. A distilled model is a separate model with fewer weights. It is often made by fine-tuning a small model on the teacher’s outputs.

§02

How it works

Distillation needs a trained teacher and a set of inputs. The teacher produces an output for each input, and the student is trained to reproduce it.

In the method Hinton, Vinyals and Dean named, the outputs are probabilities. An image model gives every possible label a probability. The wrong labels get small ones that still carry information. A BMW is rarely mistaken for a garbage truck, but far more often than for a carrot.4 Raising the temperature of the teacher’s final step spreads these probabilities out into soft targets. The student learns from them at the same temperature, alongside the correct labels.

A large language model can be distilled the same way, one token at a time. For its small Llama 3.2 models, Meta took logits, the raw scores behind a model’s probabilities, from Llama 3.1 8B and 70B. It used them as targets for each token during pretraining.5 That route needs the teacher’s scores, not just its text.

The other route uses only text. The teacher writes answers, and the student is fine-tuned on them. Kim and Rush introduced this form for translation in 2016. DeepSeek trained six Qwen and Llama models this way on 800,000 samples from R1. Reinforcement learning, which improves a model through rewards, is another way to train a small model. DeepSeek concluded that small models trained that way need enormous computing power and may not match distillation.6

§03

Origin and first known use

From Latin distillare, “to trickle down in minute drops”. English has used distillation since the late 1300s, and in chemistry it names purifying a liquid by evaporation and condensation. In machine learning the idea came before the name. In 2006 Cristian Buciluǎ, Rich Caruana and Alexandru Niculescu-Mizil called it model compression. Geoffrey Hinton, Oriol Vinyals and Jeff Dean named it distillation in a paper for a NIPS 2014 workshop, posted to arXiv in March 2015. Their method raises the temperature of a large model’s final step until its outputs are soft enough to learn from. In 2016 Yoon Kim and Alexander Rush applied it to training on a teacher’s generated text.

  1. 1300s Distillation, the act of distilling, enters English in the late 1300s, via Medieval Latin from Latin distillare, “to trickle down in minute drops”.
    Online Etymology Dictionary, “distillation” and “distill” Etymonline →: source for 1300s
  2. 2006 Buciluǎ, Caruana and Niculescu-Mizil compress large ensembles of models into small, fast neural networks, trained on a big set of data the ensemble has labeled.
    Cristian Buciluǎ, Rich Caruana and Alexandru Niculescu-Mizil, “Model Compression”, KDD 2006, Philadelphia cornell.edu →: source for 2006
  3. 2014 Hinton, Vinyals and Dean name the method distillation: a small model learns from a large model’s softened outputs. Their paper, for a NIPS 2014 workshop, is posted in March 2015.
    Geoffrey Hinton, Oriol Vinyals and Jeff Dean, “Distilling the Knowledge in a Neural Network”, NIPS 2014 Deep Learning Workshop; arXiv, 9 March 2015 arXiv 1503.02531 →: source for 2014
  4. 2016 Kim and Rush’s sequence-level version trains a translation model on the translations its teacher generates, not on its probabilities. Their best student runs 10 times faster than its teacher.
    Yoon Kim and Alexander M. Rush, “Sequence-Level Knowledge Distillation”, EMNLP 2016 arXiv 1606.07947 →: source for 2016
  5. 2019 Hugging Face distills BERT during pretraining into DistilBERT, a general-purpose model 40% smaller and 60% faster that keeps 97% of BERT’s language understanding.
    Victor Sanh, Lysandre Debut, Julien Chaumond and Thomas Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter” arXiv 1910.01108 →: source for 2019
  6. 2024 OpenAI adds Model Distillation to its API, so developers can fine-tune a cheaper model, such as GPT-4o mini, on outputs from GPT-4o or o1-preview.
    OpenAI, “Model Distillation in the API”, 1 October 2024 openai.com →: source for 2024
  7. 2025 DeepSeek releases six small models distilled from its R1 reasoning model and puts its code and models under the MIT License.
    DeepSeek, “DeepSeek-R1 Release”, 20 January 2025 api-docs.deepseek.com →: source for 2025
  8. 2026 Anthropic says DeepSeek, Moonshot and MiniMax generated over 16 million exchanges with Claude through about 24,000 fraudulent accounts, to distill its capabilities.
    Anthropic, “Detecting and preventing distillation attacks”, 23 February 2026 anthropic.com →: source for 2026
§04

In the wild

9 sightings · 2014–2026 · oldest first
  1. Paper19 Dec 2014

    “The recently proposed knowledge distillation approach is aimed at obtaining small and fast-to-execute models…”

    Romero et al., arXiv preprint arXiv 1412.6550 →: Romero et al., arXiv preprint, 19 Dec 2014Archived →: copy of Romero et al., arXiv preprint, 19 Dec 2014
  2. Paper25 Jun 2016

    “…introduce two novel sequence-level versions of knowledge distillation that further improve performance, and somewhat surprisingly, seem to eliminate the need for beam search…”

    Kim and Rush, EMNLP 2016 arXiv 1606.07947 →: Kim and Rush, EMNLP 2016, 25 Jun 2016Archived →: copy of Kim and Rush, EMNLP 2016, 25 Jun 2016
  3. Paper2 Oct 2019

    “While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase…”

    Sanh et al., Hugging Face, NeurIPS 2019 workshop paper arXiv 1910.01108 →: Sanh et al., Hugging Face, NeurIPS 2019 workshop paper, 2 Oct 2019Archived →: copy of Sanh et al., Hugging Face, NeurIPS 2019 workshop paper, 2 Oct 2019
  4. Blog14 May 2024

    “…a process called “distillation,” where the most essential knowledge and skills from a larger model are transferred to a smaller, more efficient model.”

    Demis Hassabis, Google blog blog.google →: Demis Hassabis, Google blog, 14 May 2024Archived →: copy of Demis Hassabis, Google blog, 14 May 2024
  5. Blog25 Sep 2024

    “Knowledge distillation uses a larger network to impart knowledge on a smaller network…”

    Meta, Llama 3.2 announcement ai.meta.com →: Meta, Llama 3.2 announcement, 25 Sep 2024Archived →: copy of Meta, Llama 3.2 announcement, 25 Sep 2024
  6. Blog1 Oct 2024

    “Model distillation involves fine-tuning smaller, cost-efficient models using outputs from more capable models…”

    OpenAI, product announcement openai.com →: OpenAI, product announcement, 1 Oct 2024Archived →: copy of OpenAI, product announcement, 1 Oct 2024
  7. Docs20 Jan 2025

    “API outputs can now be used for fine-tuning & distillation”

    DeepSeek, R1 release note in its API documentation api-docs.deepseek.com →: DeepSeek, R1 release note in its API documentation, 20 Jan 2025Archived →: copy of DeepSeek, R1 release note in its API documentation, 20 Jan 2025
  8. Headline29 May 2025

    “DeepSeek’s distilled new R1 AI model can run on a single GPU”

    TechCrunch techcrunch.com →: TechCrunch, 29 May 2025Archived →: copy of TechCrunch, 29 May 2025
  9. Blog23 Feb 2026

    “…frontier AI labs routinely distill their own models to create smaller, cheaper versions for their customers.”

    Anthropic, company announcement anthropic.com →: Anthropic, company announcement, 23 Feb 2026Archived →: copy of Anthropic, company announcement, 23 Feb 2026
§05

Not to be confused with

  • Quantization

    Quantization keeps the original model and every weight, and stores each weight in fewer bits. Distillation trains a separate, smaller model to imitate the larger one, so the result has fewer weights.

    See quantization →

  • Fine-tuning

    Fine-tuning is further training of an existing model on examples from any source. Distillation is defined by its targets, which come from a teacher model’s outputs. It is often carried out by fine-tuning a smaller model on them.

    See fine-tuning →

  • Pruning

    Pruning removes parts of an existing model to shrink it. Distillation trains a separate student to imitate a teacher. Meta combined the two for its small Llama 3.2 models, using distillation after pruning to recover performance.

  • Distillation in chemistry

    The chemical process purifies a liquid by evaporating and condensing it. The AI method borrowed the name. Its temperature is only a number that softens a model’s output probabilities.

§06

Questions

  • What is distillation in AI?

    Distillation is a way to build a smaller, cheaper AI model from a larger one. The large model, called the teacher, produces outputs, and the small model, the student, is trained to reproduce them. Google trained Gemini 1.5 Flash from Gemini 1.5 Pro this way. Hugging Face’s DistilBERT is 40% smaller and 60% faster than BERT and keeps 97% of its language understanding.

  • What is the difference between distillation and quantization?

    Distillation trains a new, smaller model to imitate a larger one, so the result has fewer weights. Quantization keeps the same model and all of its weights, and stores each weight in fewer bits. Pruning, a third method, removes parts of an existing model. Meta combined two of these for its small Llama 3.2 models: it pruned a larger Llama model, then used distillation to recover performance.

  • Can a company distill a competitor’s AI model?

    It depends on the provider’s terms. OpenAI’s terms of use forbid using its output to develop models that compete with OpenAI. DeepSeek took the opposite approach when it released R1 in January 2025, allowing its API outputs to be used for fine-tuning and distillation. Anthropic says labs commonly distill their own models, but reported in 2026 that three labs used about 24,000 fraudulent accounts to distill Claude.

  • Did DeepSeek distill OpenAI’s models?

    OpenAI has accused it of doing so. In February 2026 OpenAI sent House lawmakers a memo saying DeepSeek had used distillation to mimic its products. Anthropic has made a similar claim about Claude. DeepSeek’s researchers have said that R1’s success did not hinge on training with the output of rival models. DeepSeek does distill its own models, and released six smaller ones trained on samples from R1.

  • Why does distillation work?

    A trained model’s outputs say more than the right answer. An image model gives every label a probability, and the small ones show which mistakes are likely. A BMW is rarely mistaken for a garbage truck, but far more often than for a carrot. Hinton, Vinyals and Dean called these soft targets. A student trained on them learns from that extra information as well as from the correct labels.

§07

Watch

§09

Cite this entry

“Distillation.” SUPERR_dictionary, Superrduperr, 24 Sept. 2026, superrduperr.com/dictionary/distillation.

First published 24 Sep 2026 · Last reviewed 24 Sep 2026 · Report an error →