Code Language Models
Pretraining and fine-tuning language models on source code for completion, infilling, generation and repair.
Tier Grades
Tier A: Frontier AI labs
feasible
Meta, Alibaba, DeepSeek and Mistral pretrain code models on trillions of tokens (StarCoder2: up to 4.3 trillion).
Tier B: Indonesian AI labs
stretch
Pretraining on 3 to 4 trillion tokens (StarCoder2) is a stretch; fine-tuning an open code model is not; Indonesian lab capacity is not in the sources.
Tier C: Institution units such as ITB
stretch
StarCoder2 needed a 66-author open collaboration, a stretch for one institution unit.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Open 7B to 15B code models (StarCoder2, Code Llama) serve completion and code assistance proofs of concept.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeStarCoder2: 3B, 7B and 15B models on 3.3 to 4.3 trillion tokens. Code Llama: 7B, 13B, 34B and 70B, trained on 16k-token sequences.
- DataThe Stack v2 from the Software Heritage archive, 619 programming languages, 4 times larger than the first StarCoder dataset.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 66 authors on StarCoder2; 26 on Code Llama.
- Time horizonMonths.
- Shelf-life signalsGeneral LLMs absorbing code ability (IBM folds coding into Granite 4.2), Open code models from 7B to 30B as completion engines in IDEs
- Pipeline roleTier A and open collaborations pretrain code models; Tier C fine-tunes and evaluates them; Tier D deploys them as assistants in client proofs of value.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
DeepSeek-Coder-V2codeexemplar
StarCoder2-15Bcodeexemplar
Code Llamacodeexemplar
Qwen2.5-Codercodeexemplar
Mellum2.1-12B-A2.5B-Thinkingcodeproduced by
SWE-2codeproduced by
Laguna S 2.1codeproduced by
GLM-5.2LLMproduced by
North Mini Codecodeproduced by
Kimi K2.7 Codecodeproduced by
Composer 2.5codeproduced byComposer 2codeproduced by
GPT-5.3-Codexcodeproduced byQwen3-Coder-Nextcodeproduced byGPT-5.1-Codex-Maxcodeproduced by
Devstral 2 (123B, 2512)codeproduced byDevstral Small 2 (24B, 2512)codeproduced byGPT-5.1-Codexcodeproduced byGPT-5.1-Codex-Minicodeproduced by
KAT-Coder-Pro V1codeproduced by
grok-code-fast-1codeproduced byCodestral 25.08codeproduced byQwen3-Codercodeproduced by
DiffuCoder 7B Instructcodeproduced by+41
Devstral Medium 1.0codeproduced byDevstral Small 1.1codeproduced by
Seed-Coder-8B-Instructcodeproduced by
SWE-agent-LM-32Bcodeproduced by
OpenHands LM 32B v0.1codeproduced byQwen3-Coder-30B-A3B-Instructcodeproduced byQwen3-Coder-480B-A35B-Instructcodeproduced bySWE-Gym agentscodeproduced byQwen2.5-Coder 14Bcodeproduced byQwen2.5-Coder-32B-Instructcodeproduced byQwen2.5-Coder 1.5Bcodeproduced byQwen2.5-Coder 7Bcodeproduced by
Granite 8B Code Instruct 128Kcodeproduced byDeepSeek-Coder-V2-Basecodeproduced byGranite 20B Code Instruct 8Kcodeproduced byGranite 34B Code Instruct 8Kcodeproduced byGranite Codecodeproduced by
CodeGemmacodeproduced byCodeGemma 2Bcodeproduced byCodeGemma 7Bcodeproduced byCodeGemma 7B ITcodeproduced byMamba-Codestral-7B-v0.1codeproduced by
CodeGen2.5 7B multicodeproduced byCodeGen 6B multicodeproduced byCodeGencodeproduced byCodeT5 basecodeproduced byComposercodeproduced byDoubao Seed Codecodeproduced bySeed-Coder-8B-Basecodeproduced bySWE (Devin)codeproduced bySWE-1.6 previewcodeproduced bySWE-1.7codeproduced by
MAI-Code-1.1-Flashcodeproduced byCodestralcodeproduced byDevstralcodeproduced byCode Llama 13B Instructcodeproduced byCode Llama 34B Pythoncodeproduced byCode Llama 70B Instructcodeproduced byCode Llama 7Bcodeproduced bycodex-mini-latestcodeproduced byGPT-5.2-Codexcodeproduced byLinks
R&D TopicCode Language Models
ModelsQwen2.5-CoderAlibaba (Qwen), codeStarCoder2-15BBigCode (Hugging Face and ServiceNow), codeDeepSeek-Coder-V2DeepSeek, codeCode LlamaMeta, code+61 more
Qwen2.5-CoderAlibaba (Qwen), codeStarCoder2-15BBigCode (Hugging Face and ServiceNow), codeDeepSeek-Coder-V2DeepSeek, codeCode LlamaMeta, code+61 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 65
DevstralMistral AI, codeproduced byGLM-5.2Z.ai (Zhipu), LLMproduced byCode LlamaMeta, codeexemplarComposer 2.5Anysphere (Cursor), codeproduced byGPT-5.1-CodexOpenAI, codeproduced byQwen2.5-CoderAlibaba (Qwen), codeexemplarQwen3-CoderAlibaba (Qwen), codeproduced bySWE (Devin)Cognition (owns Windsurf), codeproduced bySWE-1.7Cognition (owns Windsurf), codeproduced bySWE-2Cognition (owns Windsurf), codeproduced byCodeGemmaGoogle, codeproduced byComposerAnysphere (Cursor), codeproduced by+53
Composer 2Anysphere (Cursor), codeproduced byDevstral Small 2 (24B, 2512)Mistral AI, codeproduced byGPT-5.1-Codex-MaxOpenAI, codeproduced byGPT-5.2-CodexOpenAI, codeproduced byGranite CodeIBM, codeproduced byQwen2.5-Coder-32B-InstructAlibaba (Qwen), codeproduced bySWE-1.6 previewCognition (owns Windsurf), codeproduced bySWE-agent-LM-32BSWE-bench team (Princeton/Stanford), codeproduced byCodeGenSalesforce, codeproduced byCodestralMistral AI, codeproduced byDeepSeek-Coder-V2-BaseDeepSeek, codeproduced byDevstral 2 (123B, 2512)Mistral AI, codeproduced byDevstral Medium 1.0Mistral AI, codeproduced byDevstral Small 1.1Mistral AI, codeproduced byGPT-5.1-Codex-MiniOpenAI, codeproduced byGPT-5.3-CodexOpenAI, codeproduced bygrok-code-fast-1SpaceXAI (xAI), codeproduced byKAT-Coder-Pro V1KwaiKAT, codeproduced byKimi K2.7 CodeMoonshot AI, codeproduced byMamba-Codestral-7B-v0.1Mistral AI, codeproduced byNorth Mini CodeCohere, codeproduced byOpenHands LM 32B v0.1All Hands AI, codeproduced byQwen3-Coder-30B-A3B-InstructAlibaba (Qwen), codeproduced byQwen3-Coder-480B-A35B-InstructAlibaba (Qwen), codeproduced byQwen3-Coder-NextAlibaba (Qwen), codeproduced byCode Llama 13B InstructMeta, codeproduced byCode Llama 34B PythonMeta, codeproduced byCode Llama 70B InstructMeta, codeproduced byCode Llama 7BMeta, codeproduced byCodeGemma 2BGoogle, codeproduced byCodeGemma 7BGoogle, codeproduced byCodeGemma 7B ITGoogle, codeproduced byCodeGen 6B multiSalesforce, codeproduced byCodeGen2.5 7B multiSalesforce, codeproduced byCodestral 25.08Mistral AI, codeproduced bycodex-mini-latestOpenAI, codeproduced byDeepSeek-Coder-V2DeepSeek, codeexemplarDiffuCoder 7B InstructApple, codeproduced byGranite 20B Code Instruct 8KIBM, codeproduced byGranite 34B Code Instruct 8KIBM, codeproduced byGranite 8B Code Instruct 128KIBM, codeproduced byLaguna S 2.1poolside, codeproduced byMAI-Code-1.1-FlashMicrosoft AI, codeproduced byMellum2.1-12B-A2.5B-ThinkingJetBrains, codeproduced byQwen2.5-Coder 1.5BAlibaba (Qwen), codeproduced byQwen2.5-Coder 14BAlibaba (Qwen), codeproduced byQwen2.5-Coder 7BAlibaba (Qwen), codeproduced bySeed-Coder-8B-BaseByteDance, codeproduced bySeed-Coder-8B-InstructByteDance, codeproduced bySWE-Gym agentsAll Hands AI, codeproduced byCodeT5 baseSalesforce, codeproduced byDoubao Seed CodeByteDance, codeproduced byStarCoder2-15BBigCode (Hugging Face and ServiceNow), codeexemplarPapers
- Evaluating Large Language Models Trained on CodeMark Chen et al., arXiv preprint, 2021 founding
- DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code IntelligenceDaya Guo et al., arXiv preprint, 2024 state of the art
- StarCoder 2 and The Stack v2: The Next GenerationAnton Lozhkov et al., arXiv preprint, 2024 state of the art
- Qwen2.5-Coder Technical ReportBinyuan Hui et al., arXiv preprint, 2024 state of the art
- A Survey on Large Language Models for Code GenerationJuyong Jiang et al., arXiv preprint, 2024 survey
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2402.19173 https://arxiv.org/abs/2402.19173
- arxiv.org/abs/2308.12950 https://arxiv.org/abs/2308.12950
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Code Language Models", https://aurion.semai.id/topics/code-language-models/, records of October 2026.