Regional or National Foundation Model Training (Continued Pretraining at 7B to 70B)

Pretraining or continued pretraining a mid-size model to serve a national or regional language set.

Tier Grades

Tier A: Frontier AI labs

feasible

medium confidence

Qwen3's expansion to 119 languages shows a frontier lab already covering regional languages at scale.

Tier B: Indonesian AI labs

feasible

medium confidence

Sahabat-AI 70B was trained in Indonesia by GoTo and Indosat on GPU Merdeka, although no hardware or token figures were published.

Tier C: Institution units such as ITB

stretch

high confidence

Academic anchors (1 to 8 GPUs; Pythia-1B needs 18 to 41 days on 4 A100) make 7B to 9B continued pretraining a stretch and 70B out of reach.

Tier D: Organisations such as SEMAI (PoT, PoC, PoV)

not feasible

medium confidence

The notes show only telco-scale or national-programme actors doing this, with no proof-of-concept-scale anchor.

Where this comes from

Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.

Anchors

ComputeSahabat-AI 70B trained on Indosat GPU Merdeka via Lintasarta (no hardware figures published); SEA-LION v3 built on 8B and 9B models; compute not published by either source.
DataSahabat-AI covers Bahasa Indonesia, Javanese, Sundanese, Balinese, Bataknese; SEA-LION covers 11 Southeast Asian languages; token counts not published.
Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): SEA-LION paper has 30 authors.
Time horizonRelease cadence about yearly (unverified).
Shelf-life signalsOpen-vs-closed lag of about 14 months bears on how long a regional model on an open base stays competitive, Qwen3 expanded from 29 to 119 languages, a frontier lab entering regional language coverage
Pipeline roleTier A supplies the open base models; Tier B continues pretraining them for national languages and passes the regional model to Tier C for evaluation and to Tier D for proof-of-value deployments.

Where this comes from

Team size is a proxy (author counts of exemplar papers), not a headcount.

Who and What

Exemplars

Sahabat-AI AI Singapore

SEA-LION

Observed actors

GoTo Gojek Tokopedia Indosat Ooredoo Hutchison

Indosat Ooredoo Hutchison Lintasarta AI Singapore

AI Singapore

Linked models

+71

Qwen-SEA-LION-v4.5-27B-ITLLMproduced by

Gemma 4 E2Bedge LLMused by

Qwen3.6-27Bmultimodal LLMused by

NamazuLLMproduced by

Sarvam 105BLLMproduced by

Sarvam 30BLLMproduced by

Qwen3.5-397B-A17Bmultimodal LLMused by

Solar Pro 3LLMproduced by

Kanana-2 30B A3B InstructLLMproduced by

Falcon-H1-ArabicLLMproduced by

GPT-OSS-SwallowLLMproduced by

Qwen3-SwallowLLMproduced by

Nemotron 3 SuperLLMused by

Nemotron 3 Nano 30B-A3BLLMused by

Qwen-SEA-LION-v4-8B-VLmultimodal LLMproduced by

Bielik 11B v3.0 InstructLLMproduced by

Apertus 70B Instruct (Apertus-70B-Instruct-2509)LLMproduced by

Apertus 8B Instruct (Apertus-8B-Instruct-2509)LLMproduced by

Gemma-SEA-LION-v4-27BLLMproduced by

Gemma-SEA-LION-v4-27B-ITLLMproduced by

gpt-oss-20bLLMused by

Teuken-7B v0.6LLMproduced by

A*STAR (Institute for Infocomm Research, I2R)

MERaLiON-2-3Bmultimodal LLMproduced by

MERaLiON-2multimodal LLMproduced by

MERaLiON-2-10Bmultimodal LLMproduced by

ALLaM-7B-Instruct-previewLLMproduced by

EuroLLM-9BLLMproduced by

gemma2-9b-cpt-sahabatai-v1LLMproduced by

llama3-8b-cpt-sahabatai-v1LLMproduced by

Sahabat-AI v1LLMproduced by

Aya Expanse 32BLLMproduced by

Gemma 2 2B JPN ITLLMproduced by

CALM3 22B ChatLLMproduced by

SeaLLMs v3LLMproduced by

SeaLLMs v3 1.5BLLMproduced by

SeaLLMs v3 7BLLMproduced by

Llama-3-ELYZA-JP-8BLLMproduced by

Gemma 2 9BLLMused by

KARAKURI LM 8x7B Instruct v0.1LLMproduced by

TAIDE (Trustworthy AI Dialogue Engine project, Taiwan)

Llama3-TAIDE-LX-8B-Chat-Alpha1LLMproduced by

SeaLMMM 7B v0.1multimodal LLMproduced by CendolLLMproduced by

Komodo-7BLLMproduced by

Swallow 7B InstructLLMproduced by

Stockmark-13bLLMproduced by

ViT5 baseLLMproduced by

gemma2-9b-cpt-sea-lionv3-base (SEA-LION v3)LLMproduced by

SEA-LIONLLMproduced by

SEA-LION v4.5 (Qwen) 27BLLMproduced by

Param-1 2.9BLLMproduced by

Param-1 7BLLMproduced by

JaisLLMproduced by

GemmaLLMused by

gemma2-9b-cpt-sahabatai-v1-base (Sahabat-AI v1 base)LLMproduced by

gemma2-9b-cpt-sahabatai-v1-instruct (Sahabat-AI v1 instruct)LLMproduced by

Sahabat-AILLMproduced by

GPT-OSS-Swallow-20BLLMproduced by

Medical-SwallowLLMproduced by

K-EXAONELLMproduced by

Llama 3.1 70BLLMused by

Llama 3 8BLLMused by

HyperCLOVA X SEED Think 32Breasoningproduced by

LLM-jp-4-VL-9Bmultimodal LLMproduced by

tsuzumiLLMproduced by

KrutrimLLMproduced by

SarashinaLLMproduced by

Apertus 1.0LLMproduced by

Apertus-70B-2509LLMproduced by

Apertus-8B-2509LLMproduced by

Apertus v1.5LLMproduced by

Solar-Open-100BLLMproduced by

Papers

SEA-LION: Southeast Asian Languages in One NetworkRaymond Ng et al., ACL 2025 (per arXiv comment), 2025 state of the art
Sailor: Open Language Models for South-East AsiaLongxu Dou et al., arXiv preprint, 2024 state of the art
Typhoon: Thai Large Language ModelsKunat Pipatanakul et al., arXiv preprint, 2023 state of the art
SeaLLMs -- Large Language Models for Southeast AsiaXuan-Phi Nguyen et al., ACL 2024 (per arXiv comment), 2023 state of the art
Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language ModelsNeha Sengupta et al., arXiv preprint, 2023 state of the art

Where this comes from

The papers to read first: where the direction began, its current best, and a survey when one exists.

Sources

Qwen3 Technical Report arxiv.org https://arxiv.org/abs/2505.09388
$100K or 100 Days:Trade-offs when Pre-Training with Academic Resources arxiv.org https://arxiv.org/html/2410.23261v1
Open vs. closed AI: How behind are open models? epoch.ai https://epoch.ai/blog/open-models-report
Sahabat-AI Gets Smarter: Indosat and GoTo Launch New 70 Billion Parameter Model with Multilingual Chat Service gotocompany.com https://www.gotocompany.com/en/news/press/sahabat-ai-gets-smarter-indosat-and-goto-launch-new-70-billion-parameter-model-with-multilingual-chat-service
SEA-LION: Southeast Asian Languages in One Network arxiv.org https://arxiv.org/abs/2504.05747

To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Regional or National Foundation Model Training (Continued Pretraining at 7B to 70B)", https://aurion.semai.id/topics/regional-national-foundation-model-training/, records of October 2026.

Regional or National Foundation Model Training (Continued Pretraining at 7B to 70B)

Tier Grades

Tier A: Frontier AI labs

Tier B: Indonesian AI labs

Tier C: Institution units such as ITB

Tier D: Organisations such as SEMAI (PoT, PoC, PoV)

Anchors

Who and What

Links

Models 95

Papers

Sources