Distillation of Frontier Models into Smaller Ones
Transferring capability from a large teacher model to a smaller one.
Tier Grades
Tier A: Frontier AI labs
feasible
medium confidence
Alibaba and DeepSeek distil from their own flagships, as stated in Qwen3 and R1.
Tier B: Indonesian AI labs
stretch
low confidence
The notes give no compute figure for distillation, and the observed actors own the teacher model; the notes verify no Indonesian national compute programme, BRIN facility or ITB GPU inventory (T11 Part E item 2).
Tier C: Institution units such as ITB
stretch
medium confidence
The notes give no compute figure; the closest academic anchor is the s1 fine-tune (topic 4) on an open base.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
not feasible
low confidence
The notes give nothing placing distillation at proof-of-concept scale; d would consume distilled models rather than produce them.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeNot sourced: Qwen3 states distillation from the flagship reduces the compute needed; no figures given.
- DataTeacher outputs; R1 states large-model reasoning patterns can improve smaller models.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): not sourced in the notes.
- Time horizonWeeks.
- Shelf-life signalsnot in the notes
- Pipeline roleTier A distils its flagships into small open models that Tier B and Tier C fine-tune and Tier D deploys.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
Qwen3LLMexemplar
DeepSeek-R1reasoningexemplar
Gemma-SEA-LION-v4.5-E2B-ITedge LLMused byQwen-SEA-LION-v4.5-27B-ITLLMused byDeepSeek-R1-Distill-Llama-70Breasoningproduced byDeepSeek-R1-Distill-Llama-8Breasoningproduced byDeepSeek-R1-Distill-Qwen-1.5Breasoningproduced byDeepSeek-R1-Distill-Qwen-14Breasoningproduced byDeepSeek-R1-Distill-Qwen-32Breasoningproduced byDeepSeek-R1-Distill-Qwen-7Breasoningproduced by
Llama-3.3-70B-InstructLLMused by
Nemotron-Mini-4B-Instructedge LLMused byMistral-NeMo-Minitron 8B baseLLMproduced by
Zephyr 7B betaLLMused byDistilBART CNN 12-6LLMused by
MiMo-V2.6-Pro distilled Qwen-9B variantLLMproduced byMiMo-V2.6-Pro-MOPDLLMproduced byLinks
R&D TopicDistillation of Frontier Models into Smaller Ones
ModelsQwen3Alibaba (Qwen), LLMDeepSeek-R1DeepSeek, reasoningGemma-SEA-LION-v4.5-E2B-ITAI Singapore, edge LLMQwen-SEA-LION-v4.5-27B-ITAI Singapore, LLM+13 more
Qwen3Alibaba (Qwen), LLMDeepSeek-R1DeepSeek, reasoningGemma-SEA-LION-v4.5-E2B-ITAI Singapore, edge LLMQwen-SEA-LION-v4.5-27B-ITAI Singapore, LLM+13 moreWho Runs Itnone recorded
Models 17
Qwen3Alibaba (Qwen), LLMexemplarDeepSeek-R1DeepSeek, reasoningexemplarGemma-SEA-LION-v4.5-E2B-ITAI Singapore, edge LLMused byQwen-SEA-LION-v4.5-27B-ITAI Singapore, LLMused byDeepSeek-R1-Distill-Llama-8BDeepSeek, reasoningproduced byDeepSeek-R1-Distill-Qwen-1.5BDeepSeek, reasoningproduced byDeepSeek-R1-Distill-Qwen-7BDeepSeek, reasoningproduced byDeepSeek-R1-Distill-Llama-70BDeepSeek, reasoningproduced byDeepSeek-R1-Distill-Qwen-14BDeepSeek, reasoningproduced byDeepSeek-R1-Distill-Qwen-32BDeepSeek, reasoningproduced byLlama-3.3-70B-InstructMeta, LLMused byDistilBART CNN 12-6Hugging Face, LLMused byPapers
- Distilling the Knowledge in a Neural NetworkGeoffrey Hinton et al., NIPS 2014 (per arXiv comment), 2015 founding
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighterVictor Sanh et al., NeurIPS 2019 (per arXiv comment), 2019 founding
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningDeepSeek-AI et al., Nature volume 645, pages 633-638 (2025), 2025 state of the art
- Orca: Progressive Learning from Complex Explanation Traces of GPT-4Subhabrata Mukherjee et al., arXiv preprint, 2023 state of the art
- A Survey on Knowledge Distillation of Large Language ModelsXiaohan Xu et al., arXiv preprint, 2024 survey
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- Qwen3 Technical Report arxiv.org https://arxiv.org/abs/2505.09388
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning arxiv.org https://arxiv.org/abs/2501.12948
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Distillation of Frontier Models into Smaller Ones", https://aurion.semai.id/topics/distillation-of-frontier-models/, records of October 2026.