SWE-bench Verified
Human-validated SWE-bench subset; near saturation at the top, contamination and test-design issues noted
codingcontaminated
Score in %
Definition
- Version
- not in the notes
- Category
- coding
- Maintainer
- SWE-bench team (Princeton and Stanford), with OpenAI Preparedness for the Verified subset
- Status
- contaminated
- Home page
- https://www.swebench.com/
Sources
- leaderboard.steel.dev/leaderboards/swe-bench-verified.md https://leaderboard.steel.dev/leaderboards/swe-bench-verified.md
- SWE-bench Leaderboards swebench.com https://www.swebench.com/
- deepseek-ai/DeepSeek-V4-Pro , Hugging Face huggingface.co https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro
- openai.com/index/why-we-no-longer-evaluate-swe-bench-ver... https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/
- SWE-bench Verified Leaderboard (October 2026): Top Scores benchlm.ai https://benchlm.ai/benchmarks/swe-bench-verified
- github.com/SWE-bench/SWE-bench https://github.com/SWE-bench/SWE-bench
- gigazine.net/gsc_news/en/20260429-swe-bench-verified https://gigazine.net/gsc_news/en/20260429-swe-bench-verified
Scores at a Glance
Score in %
Higher is better for this unit.
1
Claude Opus 5AnthropicVals.ai, mini-swe-agent bash-only, +/-0.76Claude Mythos 5Anthropicmax effortClaude Fable 5Anthropic5-trial average
GPT-5.6 SolOpenAImax/xhigh, third-party via Thinking Machines
DeepSeek-V4-ProDeepSeek
Gemini 3.1 Pro (preview)Google DeepMind, Google
MiniMax-M3MiniMaxMiniMax-M2.5MiniMax
Kimi K2.6Moonshot AI
GLM-5.2Z.ai (Zhipu)third-party via Thinking MachinesDeepSeek-V4-FlashDeepSeek
Qwen 3.6 PlusAlibaba (Qwen)
Hy3Tencent
MiMo-V2-ProXiaomi
Mistral Medium 3.5 (v26.04)Mistral AI
InklingThinking Machineseffort 0.99Qwen3.6-27BAlibaba (Qwen)Kimi K2.5Moonshot AIQwen3.5-397B-A17BAlibaba (Qwen)
Muse Glimmer 30BMetaQwen3.6-35B-A3BAlibaba (Qwen)Devstral 2 (123B, 2512)Mistral AI
grok-code-fast-1SpaceXAI (xAI)xAI internal harness
Nemotron 3 Ultra 550B A55BNVIDIAthird-partyQwen3-Coder-NextAlibaba (Qwen)scaffold not stated
K-EXAONE 2.0LG AI ResearchDevstral Small 2 (24B, 2512)Mistral AIDevstral Medium 1.0Mistral AIDevstral Small 1.1Mistral AIClaude 3.5 Sonnet (Oct '24)Anthropic
Granite 4.2 8BIBM
Mellum2.1-12B-A2.5B-ThinkingJetBrainsthinking mode
Sarvam 105BSarvam AI
OpenHands LM 32B v0.1All Hands AIOpenHands scaffold
Claude Opus 5AnthropicVals.ai, mini-swe-agent bash-only, +/-0.7697 %
independent2Claude Mythos 5Anthropicmax effort95.5 %
vendor-reported3Claude Fable 5Anthropic5-trial average95 %
vendor-reported4GPT-5.6 SolOpenAImax/xhigh, third-party via Thinking Machines82.2 %
independent5DeepSeek-V4-ProDeepSeek80.6 %
vendor-reported6Gemini 3.1 Pro (preview)Google DeepMind, Google80.6 %
vendor-reported7MiniMax-M3MiniMax80.5 %
vendor-reported8MiniMax-M2.5MiniMax80.2 %
vendor-reported9Kimi K2.6Moonshot AI80.2 %
vendor-reported10GLM-5.2Z.ai (Zhipu)third-party via Thinking Machines80 %
independent11DeepSeek-V4-FlashDeepSeek79 %
vendor-reported12Qwen 3.6 PlusAlibaba (Qwen)78.8 %
vendor-reported13Hy3Tencent78 %
vendor-reported14MiMo-V2-ProXiaomi78 %
vendor-reported15Mistral Medium 3.5 (v26.04)Mistral AI77.6 %
vendor-reported16InklingThinking Machineseffort 0.9977.6 %
vendor-reported17Qwen3.6-27BAlibaba (Qwen)77.2 %
vendor-reported18Kimi K2.5Moonshot AI76.8 %
vendor-reported19Qwen3.5-397B-A17BAlibaba (Qwen)76.4 %
vendor-reported20Muse Glimmer 30BMeta76 %
vendor-reported21Qwen3.6-35B-A3BAlibaba (Qwen)73.4 %
vendor-reported22Devstral 2 (123B, 2512)Mistral AI72.2 %
vendor-reported23grok-code-fast-1SpaceXAI (xAI)xAI internal harness70.8 %
vendor-reported24Nemotron 3 Ultra 550B A55BNVIDIAthird-party70.7 %
independent25Qwen3-Coder-NextAlibaba (Qwen)scaffold not stated70.6 %
vendor-reported26K-EXAONE 2.0LG AI Research68.2 %
vendor-reported27Devstral Small 2 (24B, 2512)Mistral AI68 %
vendor-reported28Devstral Medium 1.0Mistral AI61.6 %
vendor-reported29Devstral Small 1.1Mistral AI53.6 %
vendor-reported30Claude 3.5 Sonnet (Oct '24)Anthropic49 %
vendor-reported31Granite 4.2 8BIBM47.67 %
vendor-reported32Mellum2.1-12B-A2.5B-ThinkingJetBrainsthinking mode47 %
vendor-reported33Sarvam 105BSarvam AI45 %
vendor-reported34OpenHands LM 32B v0.1All Hands AIOpenHands scaffold37.2 %
vendor-reported0255075100
Score in %, Anthropic announcement
Higher is better for this unit.
1Claude Opus 4.1AnthropicNo extended thinking; bash and string-replace file editor scaffold; full 500 problemsClaude Opus 4AnthropicNo extended thinking; bash and file editing tool scaffold
Claude Opus 4.1AnthropicNo extended thinking; bash and string-replace file editor scaffold; full 500 problems74.5 %
vendor-reported2Claude Opus 4AnthropicNo extended thinking; bash and file editing tool scaffold72.5 %
vendor-reported0255075100
Where this comes from
Each bar is one model, with its most independent recorded score. Hatched bars are vendor-reported; the word beside each bar says who ran it. Settings and dates differ between rows; the Scores table below has every result.
Scores
| Model | Score | Who ran it | Setting | View or population | Date | Source |
|---|---|---|---|---|---|---|
| Claude Opus 5 | 97 % | independent | Vals.ai, mini-swe-agent bash-only, +/-0.76 | – | – | leaderboard.steel.dev |
| Claude Opus 5 | 96 % | vendor-reported | – | – | – | leaderboard.steel.dev |
| Claude Opus 5 | 96 % | vendor-reported | Self-report; benchmark contaminated, display only | rank 1, BenchLM aggregator, top score | 10 Oct 2026 | benchlm.ai |
| Claude Mythos 5 | 95.5 % | vendor-reported | max effort | – | – | leaderboard.steel.dev |
| Claude Fable 5 | 95 % | vendor-reported | 5-trial average | – | – | leaderboard.steel.dev |
| Claude Fable 5 | 93.9 % | vendor-reported | secondary report; source flags possible Mythos attribution mix-up | – | – | ecosistemastartup.com |
| GPT-5.6 Sol | 82.2 % | independent | max/xhigh, third-party via Thinking Machines | – | – | leaderboard.steel.dev |
| DeepSeek-V4-Pro | 80.6 % | vendor-reported | – | – | – | blog.buildfastwithai.com |
| Gemini 3.1 Pro (preview) | 80.6 % | vendor-reported | – | – | – | leaderboard.steel.dev |
| MiniMax-M3 | 80.5 % | vendor-reported | – | – | – | huggingface.co |
| MiniMax-M3 | 80.5 % | vendor-reported | – | – | – | huggingface.co |
| MiniMax-M2.5 | 80.2 % | vendor-reported | – | – | – | leaderboard.steel.dev |
| Kimi K2.6 | 80.2 % | vendor-reported | – | – | – | leaderboard.steel.dev |
| GLM-5.2 | 80 % | independent | third-party via Thinking Machines | – | – | leaderboard.steel.dev |
| DeepSeek-V4-Flash | 79 % | vendor-reported | – | – | – | blog.buildfastwithai.com |
| Qwen 3.6 Plus | 78.8 % | vendor-reported | – | – | – | bitdoze.com |
| Hy3 | 78 % | vendor-reported | – | – | – | huggingface.co |
| MiMo-V2-Pro | 78 % | vendor-reported | – | – | – | leaderboard.steel.dev |
| Mistral Medium 3.5 (v26.04) | 77.6 % | vendor-reported | – | – | – | huggingface.co |
| Inkling | 77.6 % | vendor-reported | effort 0.99 | – | – | leaderboard.steel.dev |
| Qwen3.6-27B | 77.2 % | vendor-reported | – | – | – | huggingface.co |
| Kimi K2.5 | 76.8 % | vendor-reported | – | – | – | leaderboard.steel.dev |
| Qwen3.5-397B-A17B | 76.4 % | vendor-reported | – | – | – | huggingface.co |
 Motif-3 | 76.2 % | vendor-reported | temperature 1.0, top-p 0.95, max sequence 262,144 | Motif-3 model card | Aug 2026 | huggingface.co |
| Muse Glimmer 30B | 76 % | vendor-reported | – | – | – | huggingface.co |
| Claude Opus 4.1 | 74.5 % | vendor-reported | No extended thinking; bash and string-replace file editor scaffold; full 500 problems | Anthropic announcement | 5 Aug 2025 | anthropic.com |
| Qwen3.6-35B-A3B | 73.4 % | vendor-reported | – | – | – | rits.shanghai.nyu.edu |
| MiMo-V2-Flash | 73.4 % | vendor-reported | – | MiMo-V2-Flash model card, post-trained model table | 16 Dec 2025 | huggingface.co |
| Claude Opus 4 | 72.5 % | vendor-reported | No extended thinking; bash and file editing tool scaffold | Anthropic announcement | 22 May 2025 | anthropic.com |
| Devstral 2 (123B, 2512) | 72.2 % | vendor-reported | – | – | – | huggingface.co |
| grok-code-fast-1 | 70.8 % | vendor-reported | xAI internal harness | – | – | verdent.ai |
| Nemotron 3 Ultra 550B A55B | 70.7 % | vendor-reported | – | – | – | huggingface.co |
| Nemotron 3 Ultra 550B A55B | 70.7 % | independent | third-party | – | – | leaderboard.steel.dev |
| Qwen3-Coder-Next | 70.6 % | vendor-reported | scaffold not stated | – | – | huggingface.co |
| DeepSeek-V3.1-Terminus | 68.4 % | vendor-reported | Model card comparison table; mode not stated | DeepSeek-V3.1-Terminus model card | 22 Sep 2025 | huggingface.co |
| K-EXAONE 2.0 | 68.2 % | vendor-reported | – | – | – | huggingface.co |
| Devstral Small 2 (24B, 2512) | 68 % | vendor-reported | – | – | – | huggingface.co |
| DeepSeek-V3.1 | 66 % | vendor-reported | Non-thinking, agent mode | DeepSeek-V3.1 model card | 21 Aug 2025 | huggingface.co |
 Trinity-Large-Thinking | 63.2 % | vendor-reported | Marked with an asterisk on the model card | Trinity-Large-Thinking model card | 1 Apr 2026 | huggingface.co |
| Devstral Medium 1.0 | 61.6 % | vendor-reported | – | – | – | mistral.ai |
 LongCat-Flash (LongCat-Flash-Chat) | 60.4 % | vendor-reported | acc | LongCat-Flash-Chat model card | Sep 2025 | huggingface.co |
| Devstral Small 1.1 | 53.6 % | vendor-reported | – | – | – | mistral.ai |
| Claude 3.5 Sonnet (Oct '24) | 49 % | vendor-reported | – | – | 22 Oct 2024 | anthropic.com |
| Granite 4.2 8B | 47.67 % | vendor-reported | – | – | – | huggingface.co |
| Mellum2.1-12B-A2.5B-Thinking | 47 % | vendor-reported | thinking mode | – | – | huggingface.co |
| Sarvam 105B | 45 % | vendor-reported | – | – | – | huggingface.co |
| OpenHands LM 32B v0.1 | 37.2 % | vendor-reported | OpenHands scaffold | – | – | huggingface.co |
Where this comes from
Scores from different settings, harnesses or dates are not directly comparable; the row says which. Sorted by score, highest first, as a reading aid, not a ranking.
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "SWE-Bench Verified", https://aurion.semai.id/benchmarks/swe-bench-verified/, records of October 2026.