Multilingual and Regional Text Encoders
Text encoders pre-trained on many languages, or on one region's languages, for cross-lingual classification, tagging and retrieval.
Tier Grades
Tier A: Frontier AI labs
feasible
Meta trained XLM-R on two terabytes of text in 100 languages.
Tier B: Indonesian AI labs
feasible
MuRIL shows a regional encoder for 17 Indian languages, a pattern an Indonesian lab can follow.
Tier C: Institution units such as ITB
feasible
mmBERT came from a 6-author academic team.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Released multilingual encoders fine-tune for Indonesian classification in proofs of concept.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeNot in the sources read: XLM-R, mmBERT and MuRIL abstracts give no training compute.
- DataXLM-R: over two terabytes of filtered CommonCrawl in 100 languages; mmBERT: 3T tokens in over 1,800 languages.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 10 authors on XLM-R; 6 on mmBERT; 14 on MuRIL.
- Time horizonMonths.
- Shelf-life signalsMultilingual embedding models covering Indonesian regional languages well
- Pipeline roleTier A and Tier C release multilingual encoders; Tier B builds Indonesian and regional ones; Tier D fine-tunes them for local text tasks.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
mmBERTlanguage encoderexemplar
MuRILlanguage encoderexemplar
XLM-RoBERTalanguage encoderexemplar
Indonesian RoBERTa baselanguage encoderexemplarmmBERT smalllanguage encoderproduced bymmBERT baselanguage encoderproduced by
CamemBERTlanguage encoderproduced byBERT base multilingual casedlanguage encoderproduced byXLM-RoBERTa baselanguage encoderproduced byXLM-RoBERTa largelanguage encoderproduced by
DistilBERT base multilingual casedlanguage encoderproduced byLinks
R&D TopicMultilingual and Regional Text Encoders
ModelsXLM-RoBERTaMeta, language encodermmBERTJohns Hopkins University (CLSP), language encoderMuRILGoogle, language encoderIndonesian RoBERTa baseFlax Community (Hugging Face community event), language encoder+7 more
XLM-RoBERTaMeta, language encodermmBERTJohns Hopkins University (CLSP), language encoderMuRILGoogle, language encoderIndonesian RoBERTa baseFlax Community (Hugging Face community event), language encoder+7 moreWho Runs Itnone recorded
Models 11
DistilBERT base multilingual casedHugging Face, language encoderproduced byIndonesian RoBERTa baseFlax Community (Hugging Face community event), language encoderexemplarmmBERTJohns Hopkins University (CLSP), language encoderexemplarXLM-RoBERTaMeta, language encoderexemplarBERT base multilingual casedGoogle, language encoderproduced bymmBERT baseJohns Hopkins University (CLSP), language encoderproduced bymmBERT smallJohns Hopkins University (CLSP), language encoderproduced byXLM-RoBERTa baseMeta, language encoderproduced byXLM-RoBERTa largeMeta, language encoderproduced byCamemBERTINRIA, language encoderproduced byMuRILGoogle, language encoderexemplarPapers
- BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingJacob Devlin et al., arXiv preprint, 2018 founding
- Cross-lingual Language Model PretrainingGuillaume Lample et al., arXiv preprint, 2019 founding
- Glot500: Scaling Multilingual Corpora and Language Models to 500 LanguagesAyyoob Imani et al., ACL 2023 (per arXiv comment), 2023 state of the art
- XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language ModelsDavis Liang et al., EMNLP 2023 (per arXiv comment), 2023 state of the art
- mT5: A massively multilingual pre-trained text-to-text transformerLinting Xue et al., arXiv preprint, 2020 state of the art
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/1911.02116 https://arxiv.org/abs/1911.02116
- arxiv.org/abs/2509.06888 https://arxiv.org/abs/2509.06888
- arxiv.org/abs/2103.10730 https://arxiv.org/abs/2103.10730
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Multilingual and Regional Text Encoders", https://aurion.semai.id/topics/multilingual-text-encoders/, records of October 2026.