Frontier Pretraining Scaling

Training a new base model at or near the top-5 compute of its release date (Epoch's frontier definition).

Tier Grades

Tier A: Frontier AI labs

feasible

high confidence

Observed actors are frontier labs, with 12 developers holding a 1e25 FLOP model by June 2025 at tens of millions of dollars per run.

Tier B: Indonesian AI labs

not feasible

high confidence

The cheapest anchor is DeepSeek-V3 at $5.576M for the final run alone, and the notes verify no Indonesian national compute programme, BRIN facility or ITB GPU inventory (T11 Part E item 2).

Tier C: Institution units such as ITB

not feasible

high confidence

Academic teams typically have 1 to 8 GPUs for days, against 2.788M H800 GPU hours for the cheapest frontier final run.

Tier D: Organisations such as SEMAI (PoT, PoC, PoV)

not feasible

medium confidence

The notes give no anchor placing proof-of-concept firms anywhere near 1e25 FLOP runs; their compute scale is not described.

Where this comes from

Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.

Anchors

Compute1e25 to 1e26 FLOP; a 1e25 run costs tens of millions of dollars; first 1e26 FLOP model Grok-3 (Feb 2025); frontier compute grows 4 to 5x per year; final-run cost grows about 2.4x per year, projected above $1B by 2027; DeepSeek-V3 final run 2.788M H800 GPU hours, $5.576M, excluding ablations.
DataDeepSeek-V3 14.8T tokens; GLM-4.5 23T tokens.
Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): not sourced in the notes. R&D staff are 29 to 49% of development cost for selected frontier models.
Time horizonMonths per run; 4 to 5x per year compute growth.
Shelf-life signalsOpen models lag closed by about 14 months weighted, Final-run figures understate total cost because ablations and staff are excluded
Pipeline roleTier A produces frontier base models and, when released as open weights, the substrate that Tier B, Tier C and Tier D adapt; the roughly 14-month open lag sets how stale that substrate is.

Where this comes from

Team size is a proxy (author counts of exemplar papers), not a headcount.

Who and What

Exemplars

Grok-3DeepSeek-V3 Z.ai (Zhipu)

GLM-4.5

Observed actors

SpaceXAI (xAI) DeepSeek

Z.ai (Zhipu) OpenAI

Google DeepMind Meta

Huawei (Huawei Cloud)12 developers with a 1e25 FLOP model by June 2025

Linked models

+109

Gemini 3.6 FlashLLMproduced by

Muse Spark 1.1LLMproduced by

GPT-5.6 LunaLLMproduced by

GPT-5.6 SolLLMproduced by

GPT-5.6 TerraLLMproduced by

Claude Fable 5LLMproduced by

Command A+ (command-a-plus-05-2026)multimodal LLMproduced by

Gemini 3.5 Flashmultimodal LLMproduced by

ERNIE 5.1LLMproduced by

Mistral Medium 3.5 (v26.04)multimodal LLMproduced by

GPT-5.5LLMproduced by

Muse SparkLLMproduced by

Qwen3.6 Max PreviewLLMproduced by

Qwen 3.6 PlusLLMproduced by

Gemini 3.1 Pro (preview)multimodal LLMproduced by

Claude Opus 4.6LLMproduced by

Grok 4.1 FastLLMproduced by

GPT-5.1LLMproduced by

Mistral Medium 3.1 (mistral-medium-2508)multimodal LLMproduced by

GPT-5LLMproduced by

Claude Opus 4.1LLMproduced by

Grok 4multimodal LLMproduced by

Seed1.6multimodal LLMproduced by

Claude Opus 4LLMproduced by

Mistral Medium 3 (mistral-medium-2505)multimodal LLMproduced by

GPT-4.1LLMproduced by

GPT-4.1 miniLLMproduced by

GPT-4.1 nanoLLMproduced by

GPT-4o (March 2025, chatgpt-4o-latest)multimodal LLMproduced by

GPT-4o (March 2025 update)multimodal LLMproduced by

ERNIE 4.5multimodal LLMproduced by

Qwen2.5-MaxLLMproduced by

Gemini 3 Promultimodal LLMproduced by

Grok 2 (Dec 2024, grok-2-1212)LLMproduced by

Claude 3.5 Sonnet (Oct '24)multimodal LLMproduced by

GPT-4o miniLLMproduced by

Claude 3 Haikumultimodal LLMproduced by

Claude 3 Opusmultimodal LLMproduced by

Claude 3 Sonnetmultimodal LLMproduced by

Mistral Large (Feb '24)LLMproduced by

Mistral Medium (Dec '23)LLMproduced by Pythia-1BLLMused by

yi-lightningLLMproduced by

Qwen3.5 Max PreviewLLMproduced by

Qwen3.7 MaxLLMproduced by

Qwen3.7 Max PreviewLLMproduced by

qwen3.7-plusLLMproduced by

qwen3.8-flashLLMproduced by

Qwen3.8-Max-PrimeLLMproduced by

qwen3.8-omni-flashmultimodal LLMproduced by

qwen3.8-omni-flash-realtimemultimodal LLMproduced by

NovaLLMproduced by

Nova 2.0 OmniLLMproduced by

Nova 2.0 Pro PreviewLLMproduced by

Nova ActLLMproduced by

Nova LiteLLMproduced by

Nova MicroLLMproduced by

Nova PremierLLMproduced by

Nova ProLLMproduced by

ClaudeLLMproduced by

Claude 3.5 HaikuLLMproduced by

Claude 3.7 SonnetLLMproduced by

Claude Haiku 4.5LLMproduced by

Claude Mythos 5LLMproduced by

Claude Mythos 5.1LLMproduced by

Claude Opus 4.5LLMproduced by

Claude Opus 4.7LLMproduced by

Claude Opus 4.8LLMproduced by

Claude Opus 5LLMproduced by

Claude 4 SonnetLLMproduced by

Claude Sonnet 4.5LLMproduced by

Claude Sonnet 4.6LLMproduced by

Claude Sonnet 5LLMproduced by

Apple server PT-MoE model (Private Cloud Compute)LLMproduced by

ERNIELLMproduced by

ERNIE 4.0LLMproduced by

ERNIE 5.0LLMproduced by

Dola-Seed 2.0 ProLLMproduced by

Seed2.1LLMproduced by

Geminimultimodal LLMproduced by

Gemini 2.0multimodal LLMproduced by

gemini-2.5-computer-use-preview-10-2025LLMproduced by

Gemini 2.5 Flashmultimodal LLMproduced by

Gemini 2.5 Flash-LiteLLMproduced by

Gemini 2.5 ProLLMproduced by

Gemini 3.1 Flash-LiteLLMproduced by

Gemini 3.5 ProLLMproduced by

Gemini 3 Flashmultimodal LLMproduced by

PaLM 2LLMproduced by

Pangu 5.5 NLPLLMproduced by

iFlytek SparkLLMproduced by

Muse Spark 1.2LLMproduced by

Muse Spark (family)LLMproduced by

GPT-3.5 Turbo (0613)LLMproduced by

GPT-4LLMproduced by

GPT-5.2LLMproduced by

GPT-5.4LLMproduced by

GPT-5.6 CyberLLMproduced by

GPT-5 miniLLMproduced by

GPT-6LLMproduced by

Gemini 3.5 Flash-Litemultimodal LLMproduced by

Gemini 3.8 Flash CyberLLMproduced by

GPT (GPT-1)LLMproduced by

GrokLLMproduced by

Grok 4.20LLMproduced by

Grok 4.5LLMproduced by Spark 4.0-2024-10-14LLMproduced by

Step 5 PreviewLLMproduced by

InklingLLMproduced by

Links

R&D TopicFrontier Pretraining Scaling

Models

DeepSeek v3DeepSeek, LLM

Grok-3SpaceXAI (xAI), LLM

GLM-4.5Z.ai (Zhipu), LLM Pythia-1BLLM+129 more

Use CasesIndustrial Copilots and Maintenance AssistantsIndustrial Clinical LLM (Frontier Proprietary)Healthcare Pharma Documents, Regulatory and Life-Science R&DHealthcare Ambient Clinical ScribingHealthcare+2 more

Who Runs It

PT Pertamina Geothermal Energy Tbk (PGE)

PT Pertamina Geothermal EnergyIn production, 14 Jul 2026

Banner HealthIn production, 11 Jan 2026 CommureIn production, 11 Jan 2026 FlatironIn production, 11 Jan 2026+39 more

Graph

Models 133

GeminiGoogle DeepMind, Google, multimodal LLMproduced by

ClaudeAnthropic, LLMproduced by

GPT-5OpenAI, LLMproduced by

GPT-5.2OpenAI, LLMproduced by

NovaAmazon, LLMproduced by

Claude Opus 4.5Anthropic, LLMproduced by

Claude Sonnet 4.5Anthropic, LLMproduced by

GrokSpaceXAI (xAI), LLMproduced by

ERNIEBaidu, LLMproduced by

GPT-6OpenAI, LLMproduced by

Claude Mythos 5Anthropic, LLMproduced by

Gemini 3.6 FlashGoogle DeepMind, LLMproduced by

+121

GPT-5.5OpenAI, LLMproduced by

Claude 3.5 Sonnet (Oct '24)Anthropic, multimodal LLMproduced by

Claude Opus 4Anthropic, LLMproduced by

Claude Opus 4.1Anthropic, LLMproduced by

Gemini 3 ProGoogle DeepMind, Google, multimodal LLMproduced by

Gemini 3.8 FlashGoogle DeepMind, LLMproduced by

GLM-4.5Z.ai (Zhipu), LLMexemplar

GPT-5.1OpenAI, LLMproduced by

GPT-6 AstraOpenAI, multimodal LLMproduced by

K-EXAONE 2.0LG AI Research, LLMproduced by

Muse Spark (family)Meta, LLMproduced by

Muse Spark 1.1Meta, LLMproduced by

Claude 3.7 SonnetAnthropic, LLMproduced by

Claude Fable 5Anthropic, LLMproduced by

Claude Fable 5.1Anthropic, multimodal LLMproduced by

Claude Mythos 5.1Anthropic, LLMproduced by

Claude Opus 4.7Anthropic, LLMproduced by

Claude Opus 4.8Anthropic, LLMproduced by

Claude Opus 5Anthropic, LLMproduced by

ERNIE 4.5Baidu, multimodal LLMproduced by

ERNIE 5.0Baidu, LLMproduced by

Gemini 2.5 FlashGoogle DeepMind, Google, multimodal LLMproduced by

Gemini 3.1 Pro (preview)Google DeepMind, Google, multimodal LLMproduced by

Gemini 3.5 FlashGoogle DeepMind, multimodal LLMproduced by

Gemini 3.7 FlashGoogle DeepMind, LLMproduced by

gemini-2.5-computer-use-preview-10-2025Google DeepMind, LLMproduced by

GPT-5.4OpenAI, LLMproduced by

GPT-5.6 SolOpenAI, LLMproduced by

GPT-6 SolOpenAI, multimodal LLMproduced by

Grok 4SpaceXAI (xAI), multimodal LLMproduced by

Grok 4.20SpaceXAI (xAI), LLMproduced by

Grok 4.5SpaceXAI (xAI), LLMproduced by

Grok 4.6SpaceXAI (xAI), LLMproduced by

Muse Spark 1.2Meta, LLMproduced by

Nova ActAmazon, LLMproduced by

PaLM 2Google, LLMproduced by

Qwen 3.6 PlusAlibaba (Qwen), LLMproduced by

Qwen3.6 Max PreviewAlibaba (Qwen), LLMproduced by

Qwen3.8 MaxAlibaba (Qwen), multimodal LLMproduced by

Qwen3.8-Flash-NextAlibaba (Qwen), multimodal LLMproduced by

A.X K2SK Telecom, LLMproduced by

Claude Haiku 4.5Anthropic, LLMproduced by

Claude Haiku 5.5Anthropic, multimodal LLMproduced by

Claude Opus 4.6Anthropic, LLMproduced by

Claude Opus 5.5Anthropic, multimodal LLMproduced by

Claude Sonnet 5Anthropic, LLMproduced by

Claude Sonnet 5.5Anthropic, multimodal LLMproduced by

DeepSeek v3DeepSeek, LLMexemplar

ERNIE 4.0Baidu, LLMproduced by

ERNIE 5.1Baidu, LLMproduced by

Gemini 2.5 ProGoogle DeepMind, LLMproduced by

Gemini 3.8 Flash CyberGoogle DeepMind, LLMproduced by

Gemini 4 ArgonGoogle DeepMind, LLMproduced by

GPT-4.1OpenAI, LLMproduced by

GPT-4.1 miniOpenAI, LLMproduced by

GPT-5 miniOpenAI, LLMproduced by

GPT-5.6 CyberOpenAI, LLMproduced by

GPT-6 LunaOpenAI, multimodal LLMproduced by

GPT-6.1 SolOpenAI, multimodal LLMproduced by

Grok 2 (Dec 2024, grok-2-1212)SpaceXAI (xAI), LLMproduced by

Grok 4.1 FastSpaceXAI (xAI), LLMproduced by

Grok 4.7SpaceXAI (xAI), multimodal LLMproduced by

Mistral Medium 3 (mistral-medium-2505)Mistral AI, multimodal LLMproduced by

Mistral Medium 3.1 (mistral-medium-2508)Mistral AI, multimodal LLMproduced by

Muse SparkMeta, LLMproduced by

Muse Spark 1.3Meta, multimodal LLMproduced by

Nova 2.0 OmniAmazon, LLMproduced by

Pangu 5.5 NLPHuawei (Huawei Cloud), LLMproduced by

Qwen2.5-MaxAlibaba (Qwen), LLMproduced by

Qwen3.7 Max PreviewAlibaba (Qwen), LLMproduced by

qwen3.8-omni-flashAlibaba (Qwen), multimodal LLMproduced by

qwen3.8-omni-flash-realtimeAlibaba (Qwen), multimodal LLMproduced by

Apple server PT-MoE model (Private Cloud Compute)Apple, LLMproduced by

Claude 3 HaikuAnthropic, multimodal LLMproduced by

Claude 3 OpusAnthropic, multimodal LLMproduced by

Claude 3 SonnetAnthropic, multimodal LLMproduced by

Claude 3.5 HaikuAnthropic, LLMproduced by

Claude 4 SonnetAnthropic, LLMproduced by

Claude Sonnet 4.6Anthropic, LLMproduced by

Gemini 2.0Google DeepMind, multimodal LLMproduced by

Gemini 2.5 Flash-LiteGoogle DeepMind, LLMproduced by

Gemini 3 FlashGoogle, multimodal LLMproduced by

Gemini 3.1 Flash-LiteGoogle DeepMind, LLMproduced by

Gemini 3.5 Flash-LiteGoogle DeepMind, multimodal LLMproduced by

Gemini 3.5 ProGoogle DeepMind, LLMproduced by

GPT-4OpenAI, LLMproduced by

GPT-4.1 nanoOpenAI, LLMproduced by

GPT-4o (March 2025 update)OpenAI, multimodal LLMproduced by

GPT-4o (March 2025, chatgpt-4o-latest)OpenAI, multimodal LLMproduced by

GPT-5.6 LunaOpenAI, LLMproduced by

GPT-5.6 TerraOpenAI, LLMproduced by

GPT-6.1 AstraOpenAI, LLMproduced by

Grok-3SpaceXAI (xAI), LLMexemplar

Hy4-previewTencent, LLMproduced by

Mistral Medium (Dec '23)Mistral AI, LLMproduced by

Mistral Medium 3.5 (v26.04)Mistral AI, multimodal LLMproduced by

Nova 2.0 Pro PreviewAmazon, LLMproduced by

Nova LiteAmazon, LLMproduced by

Nova MicroAmazon, LLMproduced by

Nova PremierAmazon, LLMproduced by

Nova ProAmazon, LLMproduced by Pythia-1BLLMused by

Qwen3.5 Max PreviewAlibaba (Qwen), LLMproduced by

qwen3.8-flashAlibaba (Qwen), LLMproduced by

Qwen3.8-Max-PrimeAlibaba (Qwen), LLMproduced by

Seed1.6ByteDance, multimodal LLMproduced by

Seed2.1ByteDance, LLMproduced by

Command A+ (command-a-plus-05-2026)Cohere, multimodal LLMproduced by

Dola-Seed 2.0 ProByteDance, LLMproduced by

GPT (GPT-1)OpenAI, LLMproduced by

GPT-3.5 Turbo (0613)OpenAI, LLMproduced by

GPT-4o miniOpenAI, LLMproduced by

iFlytek SparkiFlytek, LLMproduced by

InklingThinking Machines, LLMproduced by

Mistral Large (Feb '24)Mistral AI, LLMproduced by

Mistral Large 4 (v26.10)Mistral AI, multimodal LLMproduced by

Qwen3.7 MaxAlibaba (Qwen), LLMproduced by

qwen3.7-plusAlibaba (Qwen), LLMproduced by

Step 5 PreviewStepFun, LLMproduced by

yi-lightning01.AI, LLMproduced by Spark 4.0-2024-10-14LLMproduced by

Papers

Scaling Laws for Neural Language ModelsJared Kaplan et al., arXiv preprint, 2020 founding
Training Compute-Optimal Large Language ModelsJordan Hoffmann et al., arXiv preprint, 2022 founding
The Llama 3 Herd of ModelsAaron Grattafiori et al., arXiv preprint, 2024 state of the art
A Survey of Large Language ModelsWayne Xin Zhao et al., Frontiers of Computer Science, Volume 20, Issue 12, Article 2012627 (2026), 2023 survey

Where this comes from

The papers to read first: where the direction began, its current best, and a survey when one exists.

Sources

Training compute of frontier AI models grows by 4-5x per year epoch.ai https://epoch.ai/blog/training-compute-of-frontier-ai-models-grows-by-4-5x-per-year
How much does it cost to train frontier AI models? epoch.ai https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models
Over 30 AI models have been trained at the scale of GPT-4 epoch.ai https://epoch.ai/data-insights/models-over-1e25-flop
How many AI models will exceed compute thresholds? epoch.ai https://epoch.ai/tools/model-counts
Data on AI models epoch.ai https://epoch.ai/data/notable-ai-models
DeepSeek-V3 Technical Report arxiv.org https://arxiv.org/html/2412.19437v1
GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models arxiv.org https://arxiv.org/abs/2508.06471
Open vs. closed AI: How behind are open models? epoch.ai https://epoch.ai/blog/open-models-report

To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Frontier Pretraining Scaling", https://aurion.semai.id/topics/frontier-pretraining-scaling/, records of October 2026.