Pretrained Bidirectional Text Encoders
Pretraining BERT-style encoder-only models that read text in both directions, used for classification, entity extraction and as the base of embedders.
Tier Grades
Tier A: Frontier AI labs
feasible
Encoder pretraining at 2 trillion tokens (ModernBERT) is small next to frontier LLM runs.
Tier B: Indonesian AI labs
feasible
IndoBERT shows Indonesian encoders trained on Indo4B with a 12-task benchmark (IndoNLU).
Tier C: Institution units such as ITB
feasible
IndoNLU was produced by an 11-author academic and industry team, the scale of an institution unit.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Encoders such as IndoBERT run on common GPUs for classification and extraction in proofs of concept (ModernBERT targets common GPUs).
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeModernBERT: trained on 2 trillion tokens with a native 8,192-token sequence length; designed for inference on common GPUs.
- DataIndoBERT: trained on Indo4B, a large clean Indonesian dataset from social media, blogs, news and websites, with the 12-task IndoNLU benchmark.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 14 authors on ModernBERT; 11 on IndoNLU.
- Time horizonOne to two years.
- Shelf-life signalsEncoder refreshes with long context (ModernBERT 8,192 tokens) and regional retrains (SEA-LION ModernBERT), Encoders kept in production for their performance-size trade-off against decoder LLMs
- Pipeline roleTier B and Tier C pretrain Indonesian encoders and benchmarks; Tier D fine-tunes them for classification and extraction in proofs of value; Tier A supplies modern recipes.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
SEA-LION ModernBERT 600Membeddingexemplar
Ettinlanguage encoderexemplar
DeBERTalanguage encoderexemplar
ELECTRAlanguage encoderexemplar
RoBERTalanguage encoderexemplarBERTlanguage encoderexemplarIndoBERTembeddingexemplarmDeBERTa-v3-baseembeddingexemplar
ModernBERTembeddingexemplarSEA-LION ModernBERT 300Membeddingproduced bySEA-LION ModernBERT-Embedding 300Membeddingproduced bySEA-LION ModernBERT-Embedding 600Membeddingproduced by
LFM2.5-Encoder 230Membeddingproduced byLFM2.5-Encoder 350Membeddingproduced byEttin encoder 1Blanguage encoderproduced byEttin encoder 150Mlanguage encoderproduced bymDeBERTa-v3-base-xnli-multilingual-nli-2mil7embeddingproduced byDeBERTa-v3 baselanguage encoderproduced byDeBERTa-v3 largelanguage encoderproduced byMPNet baselanguage encoderproduced byIndoBERT baseembeddingproduced byIndoBERT largeembeddingproduced byIndoBERT liteembeddingproduced byALBERTlanguage encoderproduced by+11
XLNetlanguage encoderproduced by
ELMoembeddingproduced byBERT base uncasedlanguage encoderproduced byBERT large uncasedlanguage encoderproduced byRoBERTa baselanguage encoderproduced byRoBERTa largelanguage encoderproduced byALBERT base v2language encoderproduced byALBERT xxlarge v2language encoderproduced byELECTRA small discriminatorlanguage encoderproduced byELECTRA base discriminatorlanguage encoderproduced byELECTRA large discriminatorlanguage encoderproduced byLinks
R&D TopicPretrained Bidirectional Text Encoders
ModelsBERTGoogle, language encoderRoBERTaMeta, language encoderELECTRAGoogle, language encoderDeBERTaMicrosoft AI, language encoder+31 more
BERTGoogle, language encoderRoBERTaMeta, language encoderELECTRAGoogle, language encoderDeBERTaMicrosoft AI, language encoder+31 moreUse CasesIndonesian NLU (Classification, NER)Public SectorEmbeddings and Retrieval for SEA LanguagesPublic Sector
Who Runs Itnone recorded
Models 35
IndoBERTIndoNLP, embeddingexemplarSEA-LION ModernBERT 300MAI Singapore, embeddingproduced bySEA-LION ModernBERT 600MAI Singapore, embeddingexemplarSEA-LION ModernBERT-Embedding 300MAI Singapore, embeddingproduced bySEA-LION ModernBERT-Embedding 600MAI Singapore, embeddingproduced byALBERTGoogle, language encoderproduced byBERTGoogle, language encoderexemplarELECTRAGoogle, language encoderexemplarIndoBERT baseIndoNLP, embeddingproduced byIndoBERT largeIndoNLP, embeddingproduced byIndoBERT liteIndoNLP, embeddingproduced byModernBERTAnswer.AI, LightOn, embeddingexemplar
SEA-LION ModernBERT 300MAI Singapore, embeddingproduced bySEA-LION ModernBERT 600MAI Singapore, embeddingexemplarSEA-LION ModernBERT-Embedding 300MAI Singapore, embeddingproduced bySEA-LION ModernBERT-Embedding 600MAI Singapore, embeddingproduced byALBERTGoogle, language encoderproduced byBERTGoogle, language encoderexemplarELECTRAGoogle, language encoderexemplarIndoBERT baseIndoNLP, embeddingproduced byIndoBERT largeIndoNLP, embeddingproduced byIndoBERT liteIndoNLP, embeddingproduced byModernBERTAnswer.AI, LightOn, embeddingexemplar+23
ALBERT base v2Google, language encoderproduced byDeBERTaMicrosoft AI, language encoderexemplarEttinJohns Hopkins University (CLSP), language encoderexemplarRoBERTaMeta, language encoderexemplarALBERT xxlarge v2Google, language encoderproduced byBERT base uncasedGoogle, language encoderproduced byBERT large uncasedGoogle, language encoderproduced byDeBERTa-v3 baseMicrosoft AI, language encoderproduced byDeBERTa-v3 largeMicrosoft AI, language encoderproduced byELECTRA base discriminatorGoogle, language encoderproduced byELECTRA large discriminatorGoogle, language encoderproduced byELECTRA small discriminatorGoogle, language encoderproduced byEttin encoder 150MJohns Hopkins University (CLSP), language encoderproduced byEttin encoder 1BJohns Hopkins University (CLSP), language encoderproduced byLFM2.5-Encoder 230MLiquid AI, embeddingproduced byLFM2.5-Encoder 350MLiquid AI, embeddingproduced bymDeBERTa-v3-baseMicrosoft AI, embeddingexemplarmDeBERTa-v3-base-xnli-multilingual-nli-2mil7MoritzLaurer, embeddingproduced byRoBERTa baseMeta, language encoderproduced byRoBERTa largeMeta, language encoderproduced byXLNetGoogle, Carnegie Mellon University, language encoderproduced byELMoAllen AI, embeddingproduced byMPNet baseMicrosoft AI, language encoderproduced byPapers
- BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingJacob Devlin et al., arXiv preprint, 2018 founding
- NeoBERT: A Next-Generation BERTLola Le Breton et al., arXiv preprint, 2025 state of the art
- Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and InferenceBenjamin Warner et al., arXiv preprint, 2024 state of the art
- ELECTRA: Pre-training Text Encoders as Discriminators Rather Than GeneratorsKevin Clark et al., ICLR 2020 (per arXiv comment), 2020 state of the art
- DeBERTa: Decoding-enhanced BERT with Disentangled AttentionPengcheng He et al., arXiv preprint, 2020 state of the art
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2412.13663 https://arxiv.org/abs/2412.13663
- arxiv.org/abs/2009.05387 https://arxiv.org/abs/2009.05387
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Pretrained Bidirectional Text Encoders", https://aurion.semai.id/topics/pretrained-bidirectional-text-encoders/, records of October 2026.