Text-to-Text Pretraining and Summarisation
Encoder-decoder models pretrained to map any text task to text, with objectives tailored to abstractive summarisation.
Tier Grades
Tier A: Frontier AI labs
feasible
Google and Meta built T5, PEGASUS and BART.
Tier B: Indonesian AI labs
feasible
Fine-tuning T5-size models for Indonesian summarisation is already done (an Indonesian T5 summariser is in AURION).
Tier C: Institution units such as ITB
feasible
PEGASUS came from a 4-author team.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Summarisers of T5 or BART size run cheaply for document summarisation proofs of concept.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeT5 compares pretraining objectives, architectures and data sets systematically and combines the findings with scale (the largest release is T5 11B, per its AURION record); compute is not in the abstract.
- DataT5: Colossal Clean Crawled Corpus (C4). PEGASUS: gap-sentence pretraining evaluated on 12 summarisation datasets across news, science, emails, patents and bills.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 9 authors on T5; 4 on PEGASUS.
- Time horizonYears (mature direction).
- Shelf-life signalsDecoder-only LLMs taking over summarisation, Encoder-decoder revivals adapted from decoder models (T5Gemma)
- Pipeline roleTier A publishes pretraining; Tier B and Tier C fine-tune for Indonesian; Tier D summarises client documents.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
Flan-T5LLMexemplarPEGASUSLLMexemplar
BARTLLMexemplarT5LLMexemplarT5Gemma 2multimodal LLMproduced byT5Gemma 2 270M-270Mmultimodal LLMproduced byT5Gemma 2 4B-4Bmultimodal LLMproduced byT5Gemma 2B-2B UL2LLMproduced byT5Gemma 9B-9B UL2LLMproduced byT5GemmaLLMproduced byCendolLLMproduced by
Aya 101LLMproduced byUMT5LLMproduced byFlan-T5 baseLLMproduced byFlan-T5 largeLLMproduced byFlan-T5 XXLLLMproduced by
AlexaTM 20BLLMproduced by
ViT5 baseLLMproduced byLongT5LLMproduced by
DistilBART CNN 12-6LLMproduced byBigBird-PEGASUS large arXivLLMproduced bymT5LLMproduced by
LED base 16384LLMproduced byT5 base Indonesian summarizationLLMproduced byLinks
R&D TopicText-to-Text Pretraining and Summarisation
Use Casesnone recorded
Who Runs Itnone recorded
Models 32
CendolIndoNLP, LLMproduced byBARTMeta, LLMexemplarFlan-T5Google, LLMexemplarT5Google, LLMexemplarFlan-T5 baseGoogle, LLMproduced byPEGASUSGoogle, LLMexemplarT5GemmaGoogle, LLMproduced byAya 101Cohere Labs, LLMproduced byBigBird-PEGASUS large arXivGoogle, LLMproduced byDistilBART CNN 12-6Hugging Face, LLMproduced byFlan-T5 largeGoogle, LLMproduced byFlan-T5 XXLGoogle, LLMproduced by
BARTMeta, LLMexemplarFlan-T5Google, LLMexemplarT5Google, LLMexemplarFlan-T5 baseGoogle, LLMproduced byPEGASUSGoogle, LLMexemplarT5GemmaGoogle, LLMproduced byAya 101Cohere Labs, LLMproduced byBigBird-PEGASUS large arXivGoogle, LLMproduced byDistilBART CNN 12-6Hugging Face, LLMproduced byFlan-T5 largeGoogle, LLMproduced byFlan-T5 XXLGoogle, LLMproduced by+20
LED base 16384Allen AI, LLMproduced byT5 base Indonesian summarizationCahya Wirawan, LLMproduced byT5Gemma 2Google, multimodal LLMproduced byT5Gemma 2B-2B UL2Google, LLMproduced byT5Gemma 9B-9B UL2Google, LLMproduced byViT5 baseVietAI, LLMproduced byBART largeMeta, LLMproduced byBART large CNNMeta, LLMproduced bymT5Google, LLMproduced byPEGASUS largeGoogle, LLMproduced byPEGASUS XSumGoogle, LLMproduced byT5 11BGoogle, LLMproduced byT5 baseGoogle, LLMproduced byT5 largeGoogle, LLMproduced byT5 smallGoogle, LLMproduced byT5Gemma 2 270M-270MGoogle, multimodal LLMproduced byT5Gemma 2 4B-4BGoogle, multimodal LLMproduced byUMT5Google, LLMproduced byAlexaTM 20BAmazon, LLMproduced byLongT5Google, LLMproduced byPapers
- Get To The Point: Summarization with Pointer-Generator NetworksAbigail See et al., arXiv preprint, 2017 founding
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text TransformerColin Raffel et al., arXiv preprint, 2019 founding
- Scaling Instruction-Finetuned Language ModelsHyung Won Chung et al., arXiv preprint, 2022 state of the art
- UL2: Unifying Language Learning ParadigmsYi Tay et al., arXiv preprint, 2022 state of the art
- Pretrained Language Models for Text Generation: A SurveyJunyi Li et al., IJCAI 2021 (per arXiv comment), 2021 survey
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/1910.10683 https://arxiv.org/abs/1910.10683
- arxiv.org/abs/1912.08777 https://arxiv.org/abs/1912.08777
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Text-to-Text Pretraining and Summarisation", https://aurion.semai.id/topics/text-to-text-pretraining-and-summarisation/, records of October 2026.