MMLU-Pro
10-option multiple-choice questions across 14 domains; a harder version of MMLU
knowledge and reasoningcrowded
Score in %
Definition
- Version
- not in the notes
- Category
- knowledge and reasoning
- Maintainer
- TIGER-Lab (U. Waterloo et al.), Canada
- Status
- crowded
- Home page
- https://arxiv.org/abs/2406.01574
Sources
- deepseek-ai/DeepSeek-V4-Pro , Hugging Face huggingface.co https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark arxiv.org https://arxiv.org/abs/2406.01574
- MMLU-Pro Leaderboard & Scores , October 2026 benchlm.ai https://www.benchlm.ai/benchmarks/mmlu-pro
Scores at a Glance
Score in %
Higher is better for this unit.
1
Qwen3.5-397B-A17BAlibaba (Qwen)
DeepSeek-V4-ProDeepSeek
Nemotron 3 Ultra 550B A55BNVIDIAQwen3.6-27BAlibaba (Qwen)
Solar Open 2Upstage
Gemma 4 31BGoogle, Google DeepMind
K-EXAONE 2.0LG AI ResearchQwen3.5-9BAlibaba (Qwen)Not independently verified
Sarvam 105BSarvam AIQwen3.5-4BAlibaba (Qwen)Not independently verified
Granite 4.2 8BIBM
Phi-4-mini-instructMicrosoft AI
Jamba 1.5 MiniAI21 LabsCoT
Command R (Mar '24)Cohere, Cohere LabsTIGER-Lab MMLU-Pro as quoted on the model card
Qwen3.5-397B-A17BAlibaba (Qwen)87.8 %
vendor-reported2DeepSeek-V4-ProDeepSeek87.5 %
vendor-reported3Nemotron 3 Ultra 550B A55BNVIDIA86.8 %
vendor-reported4Qwen3.6-27BAlibaba (Qwen)86.2 %
vendor-reported5Solar Open 2Upstage86.2 %
vendor-reported6Gemma 4 31BGoogle, Google DeepMind85.2 %
vendor-reported7K-EXAONE 2.0LG AI Research83.5 %
vendor-reported8Qwen3.5-9BAlibaba (Qwen)Not independently verified82.5 %
vendor-reported9Sarvam 105BSarvam AI81.7 %
vendor-reported10Qwen3.5-4BAlibaba (Qwen)Not independently verified79.1 %
vendor-reported11Granite 4.2 8BIBM74.04 %
vendor-reported12Phi-4-mini-instructMicrosoft AI52.8 %
vendor-reported13Jamba 1.5 MiniAI21 LabsCoT42.5 %
vendor-reported14Command R (Mar '24)Cohere, Cohere LabsTIGER-Lab MMLU-Pro as quoted on the model card37.9 %
vendor-reported0255075100
Where this comes from
Each bar is one model, with its most independent recorded score. Hatched bars are vendor-reported; the word beside each bar says who ran it. Settings and dates differ between rows; the Scores table below has every result.
Scores
| Model | Score | Who ran it | Setting | View or population | Date | Source |
|---|---|---|---|---|---|---|
| Qwen3.7 Max | 89.6 % | vendor-reported | – | rank 1, BenchLM aggregator, top score | 10 Oct 2026 | benchlm.ai |
| Qwen3.5-397B-A17B | 87.8 % | vendor-reported | – | – | – | huggingface.co |
| DeepSeek-V4-Pro | 87.5 % | vendor-reported | – | – | – | huggingface.co |
| Nemotron 3 Ultra 550B A55B | 86.8 % | vendor-reported | – | – | – | huggingface.co |
| Qwen3.6-27B | 86.2 % | vendor-reported | – | – | – | huggingface.co |
| Solar Open 2 | 86.2 % | vendor-reported | – | – | – | huggingface.co |
| Gemma 4 31B | 85.2 % | vendor-reported | – | – | Apr 2026 | huggingface.co |
| DeepSeek-V3.1-Terminus | 85 % | vendor-reported | Model card comparison table; mode not stated | DeepSeek-V3.1-Terminus model card | 22 Sep 2025 | huggingface.co |
 MiMo-V2-Flash | 84.9 % | vendor-reported | – | MiMo-V2-Flash model card, post-trained model table | 16 Dec 2025 | huggingface.co |
| DeepSeek-V3.1 | 84.8 % | vendor-reported | Thinking mode, EM | DeepSeek-V3.1 model card | 21 Aug 2025 | huggingface.co |
| K-EXAONE 2.0 | 83.5 % | vendor-reported | – | – | – | huggingface.co |
 Trinity-Large-Thinking | 83.4 % | vendor-reported | – | Trinity-Large-Thinking model card | 1 Apr 2026 | huggingface.co |
 LongCat-Flash (LongCat-Flash-Chat) | 82.68 % | vendor-reported | acc | LongCat-Flash-Chat model card | Sep 2025 | huggingface.co |
| Qwen3.5-9B | 82.5 % | vendor-reported | Not independently verified | – | Mar 2026 | rits.shanghai.nyu.edu |
| EXAONE 4.0 (32B) | 81.8 % | vendor-reported | Reasoning mode | EXAONE 4.0 model card | Jul 2025 | huggingface.co |
| Sarvam 105B | 81.7 % | vendor-reported | – | – | – | huggingface.co |
| Qwen3.5-4B | 79.1 % | vendor-reported | Not independently verified | – | Mar 2026 | rits.shanghai.nyu.edu |
| Granite 4.2 8B | 74.04 % | vendor-reported | – | – | – | huggingface.co |
 Reka Flash 3 | 65 % | vendor-reported | – | Reka Flash 3 announcement | Mar 2025 | reka.ai |
| Phi-4-mini-instruct | 52.8 % | vendor-reported | – | – | – | huggingface.co |
| Granite-4.0-H-Micro | 43.48 % | vendor-reported | 5-shot, CoT | Granite-4.0-H-Micro model card | 2 Oct 2025 | huggingface.co |
| Jamba 1.5 Mini | 42.5 % | vendor-reported | CoT | – | Aug 2024 | huggingface.co |
| Command R (Mar '24) | 37.9 % | vendor-reported | TIGER-Lab MMLU-Pro as quoted on the model card | – | – | huggingface.co |
Where this comes from
Scores from different settings, harnesses or dates are not directly comparable; the row says which. Sorted by score, highest first, as a reading aid, not a ranking.
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "MMLU-Pro", https://aurion.semai.id/benchmarks/mmlu-pro/, records of October 2026.