Efficiency Architectures (MoE, Sparse Attention, Long Context)

Reducing training and inference cost per capability.

Tier Grades

Tier A: Frontier AI labs

feasible

high confidence

DeepSeek and OpenAI are observed shipping sparse attention and MoE.

Tier B: Indonesian AI labs

stretch

medium confidence

New architectures need pretraining-scale runs, though methods like YaRN are cheap; the notes verify no Indonesian national compute programme, BRIN facility or ITB GPU inventory (T11 Part E item 2).

Tier C: Institution units such as ITB

feasible

high confidence

Academic groups produced YaRN at 10x fewer tokens than prior methods.

Tier D: Organisations such as SEMAI (PoT, PoC, PoV)

stretch

low confidence

The notes give no proof-of-concept anchor; context extension on open models may fit, new architectures do not.

Where this comes from

Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.

Anchors

ComputeDeepSeek sparse attention with over 50% API price cut; YaRN context extension with 10x fewer tokens and 2.5x fewer steps than prior methods; gpt-oss MoE 5.1B of 117B active.
DataYaRN needs 10x fewer tokens than prior methods.
Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): not sourced in the notes.
Time horizonMonths.
Shelf-life signalsFixed-capability inference price falls 9x to 900x per year
Pipeline roleTier A and Tier C produce efficiency methods that lower the cost of everything Tier B and Tier D run; Tier D benefits mainly as a user.

Where this comes from

Team size is a proxy (author counts of exemplar papers), not a headcount.

Who and What

Exemplars

DeepSeek-V3.2-ExpYaRNgpt-oss MoE

Observed actors

DeepSeekAcademic groups OpenAI

OpenAI

Linked models

+80

Step-3.7-Flashmultimodal LLMproduced by

MiniMax-M3multimodal LLMproduced by

Olmo-Hybrid-7BLLMproduced by

DeepSeek-V4-FlashLLMproduced by

DeepSeek-V4-ProLLMproduced by

DeepSeek V4LLMproduced by

Gemma 4 26B-A4Bmultimodal LLMproduced by

Qwen 3.6 PlusLLMproduced by

Qwen3.5LLMproduced by

Qwen3.5-122B-A10Bmultimodal LLMproduced by

Qwen3.5-397B-A17Bmultimodal LLMproduced by

Solar Pro 3LLMproduced by

Kanana-2 30B A3B InstructLLMproduced by

Falcon-H1-ArabicLLMproduced by

Falcon-H1R-7Breasoningproduced by

Kimi K2.5multimodal LLMproduced by

Nemotron 3.5 Lightning 30B-A3BLLMproduced by

Nemotron 3 SuperLLMproduced by

MiMo-V2-FlashLLMproduced by

Nemotron 3LLMproduced by

Mistral Large 3 (2512)multimodal LLMproduced by

Ling-1TLLMproduced by

Jamba Reasoning 3Breasoningproduced by

Granite-4.0-H-Microedge LLMproduced by

Ling-flash-2.0LLMproduced by

Qwen3-Next-80B-A3B-InstructLLMproduced by

LongCat-Flash (LongCat-Flash-Chat)LLMproduced by

DeepSeek-V3.1LLMproduced by

NVIDIA Nemotron Nano 9B v2reasoningproduced by

GLM-4.5LLMproduced by

gpt-oss-120bLLMproduced by

gpt-oss-20bLLMproduced by

Qwen3-235B-A22BLLMproduced by

Qwen3-30B-A3BLLMproduced by

Granite 3.3 8B InstructLLMproduced by

Kimi-VLmultimodal LLMproduced by

Kimi-VL-A3B-Instructmultimodal LLMproduced by

Kimi-VL-A3B-Thinkingmultimodal LLMproduced by

Nemotron-H 56B baseLLMproduced by

Llama 4LLMproduced by

Llama 4 Maverickmultimodal LLMproduced by

Llama 4 Scoutmultimodal LLMproduced by

Nemotron-HLLMproduced by

Nemotron-H 8B baseLLMproduced by

Granite 3.2 8B InstructLLMproduced by

Qwen2.5-MaxLLMproduced by

DeepSeek-V2.5 (Dec 2024, DeepSeek-V2.5-1210)LLMproduced by

Granite 3.1 8B InstructLLMproduced by

SANAimage generationproduced by

SANA 1.6Bimage generationproduced by

SANA 0.6Bimage generationproduced by

Granite 3LLMproduced by

Granite 3.0 8B InstructLLMproduced by

Jamba 1.5LLMproduced by

Jamba 1.5 LargeLLMproduced by

Jamba 1.5 MiniLLMproduced by

KARAKURI LM 8x7B Instruct v0.1LLMproduced by

RecurrentGemma 9BLLMproduced by

DeepSeek-V2LLMproduced by

Snowflake Arctic InstructLLMproduced by

Mixtral 8x22B InstructLLMproduced by

RecurrentGemmaLLMproduced by

RecurrentGemma 2BLLMproduced by

DBRXLLMproduced by

Mamba-Codestral-7B-v0.1codeproduced by

Mixtral 8x7B Instruct v0.1LLMproduced by

Mixtral 8x7B v0.1LLMproduced by

DeciDiffusion 1.0image generationproduced by

BigBirdlanguage encoderproduced by

BigBird-PEGASUS large arXivLLMproduced by

Longformerlanguage encoderproduced by

Transformer-XLLLMproduced by

Longformer base 4096language encoderproduced by

Jamba 1.7 LargeLLMproduced by

Apple server PT-MoE model (Private Cloud Compute)LLMproduced by

Pangu 5.5 NLPLLMproduced by

Mixtral-8x22B-v0.1LLMproduced by

LED base 16384LLMproduced by

Falcon-H1-7B-BaseLLMproduced by

Palmyra X5LLMproduced by

Papers

Mamba: Linear-Time Sequence Modeling with Selective State SpacesAlbert Gu et al., arXiv preprint, 2023 founding
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao et al., arXiv preprint, 2022 founding
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space DualityTri Dao et al., ICML 2024 (per arXiv comment), 2024 state of the art
Mixtral of ExpertsAlbert Q. Jiang et al., arXiv preprint, 2024 state of the art
Efficient Transformers: A SurveyYi Tay et al., arXiv preprint, 2020 survey

Where this comes from

The papers to read first: where the direction began, its current best, and a survey when one exists.

Sources

openai.com/index/introducing-gpt-oss https://openai.com/index/introducing-gpt-oss/
Introducing DeepSeek-V3.2-Exp | DeepSeek API Docs api-docs.deepseek.com https://api-docs.deepseek.com/news/news250929
YaRN: Efficient Context Window Extension of Large Language Models arxiv.org https://arxiv.org/abs/2309.00071

To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Efficiency Architectures (MoE, Sparse Attention, Long Context)", https://aurion.semai.id/topics/efficiency-architectures/, records of October 2026.

Efficiency Architectures (MoE, Sparse Attention, Long Context)

Tier Grades

Tier A: Frontier AI labs

Tier B: Indonesian AI labs

Tier C: Institution units such as ITB

Tier D: Organisations such as SEMAI (PoT, PoC, PoV)

Anchors

Who and What

Links

Models 104

Papers

Sources