Compact Text Encoders Through Distillation
Shrinking pre-trained text encoders through knowledge distillation and bottleneck designs so they run fast on servers and phones.
Tier Grades
Tier A: Frontier AI labs
feasible
Google and Microsoft built MobileBERT and MiniLM.
Tier B: Indonesian AI labs
feasible
Distilling an encoder is cheaper than pre-training it, as DistilBERT shows.
Tier C: Institution units such as ITB
feasible
DistilBERT came from a 4-author team.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
A 62 ms on-phone encoder fits on-device classification pilots.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeDistilBERT is 40% smaller and 60% faster while keeping 97% of BERT's language understanding; MobileBERT runs at 62 ms on a Pixel 4.
- DataNot in the sources read: DistilBERT, MobileBERT and MiniLM abstracts do not state pre-training data size.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 4 authors on DistilBERT; 6 on MiniLM.
- Time horizonWeeks to months.
- Shelf-life signalsSmall decoder models overtaking distilled encoders on device
- Pipeline roleTier A and Tier C publish distillation methods; Tier B distils Indonesian encoders; Tier D deploys them on devices and low-cost servers.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
MobileBERTlanguage encoderexemplar
MiniLMlanguage encoderexemplar
DistilBERTlanguage encoderexemplarALBERTlanguage encoderproduced byDistilBERT base uncasedlanguage encoderproduced byDistilBERT base multilingual casedlanguage encoderproduced byDistilRoBERTa baselanguage encoderproduced byALBERT base v2language encoderproduced byLinks
R&D TopicCompact Text Encoders Through Distillation
ModelsDistilBERTHugging Face, language encoderMobileBERTGoogle, language encoderMiniLMMicrosoft AI, language encoderDistilBERT base uncasedHugging Face, language encoder+4 more
DistilBERTHugging Face, language encoderMobileBERTGoogle, language encoderMiniLMMicrosoft AI, language encoderDistilBERT base uncasedHugging Face, language encoder+4 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 8
ALBERTGoogle, language encoderproduced byDistilBERTHugging Face, language encoderexemplarALBERT base v2Google, language encoderproduced byDistilBERT base multilingual casedHugging Face, language encoderproduced byDistilBERT base uncasedHugging Face, language encoderproduced byDistilRoBERTa baseHugging Face, language encoderproduced byMiniLMMicrosoft AI, language encoderexemplarMobileBERTGoogle, language encoderexemplarPapers
- Distilling the Knowledge in a Neural NetworkGeoffrey Hinton et al., NIPS 2014 (per arXiv comment), 2015 founding
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained TransformersWenhui Wang et al., arXiv preprint, 2020 state of the art
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited DevicesZhiqing Sun et al., ACL 2020 (per arXiv comment), 2020 state of the art
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighterVictor Sanh et al., NeurIPS 2019 (per arXiv comment), 2019 state of the art
- TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao et al., EMNLP 2020 (per arXiv comment), 2019 state of the art
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/1910.01108 https://arxiv.org/abs/1910.01108
- arxiv.org/abs/2004.02984 https://arxiv.org/abs/2004.02984
- arxiv.org/abs/2002.10957 https://arxiv.org/abs/2002.10957
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Compact Text Encoders Through Distillation", https://aurion.semai.id/topics/compact-encoders-distillation/, records of October 2026.