HumanEval+
Extended HumanEval tests
codingsaturated
Definition
- Version
- not in the notes
- Category
- coding
- Maintainer
- EvalPlus
- Status
- saturated
- Home page
- not in the notes
Sources
- bigcode/starcoder2-15b , Hugging Face huggingface.co https://huggingface.co/bigcode/starcoder2-15b
- huggingface.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking https://huggingface.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking
Scores at a Glance
Score in %
Higher is better for this unit.
1
Mellum2.1-12B-A2.5B-ThinkingJetBrainsthinking mode
StarCoder2-15BBigCode (Hugging Face and ServiceNow)
Mellum2.1-12B-A2.5B-ThinkingJetBrainsthinking mode91.5 %
vendor-reported2StarCoder2-15BBigCode (Hugging Face and ServiceNow)37.8 %
vendor-reported0255075100
Where this comes from
Each bar is one model, with its most independent recorded score. Hatched bars are vendor-reported; the word beside each bar says who ran it. Settings and dates differ between rows; the Scores table below has every result.
Scores
| Model | Score | Who ran it | Setting | View or population | Date | Source |
|---|---|---|---|---|---|---|
| Mellum2.1-12B-A2.5B-Thinking | 91.5 % | vendor-reported | thinking mode | – | – | huggingface.co |
 Olmo 3.1 32B Instruct | 86.7 % | vendor-reported | – | Olmo 3.1 32B Instruct model card | Dec 2025 | huggingface.co |
| StarCoder2-15B | 37.8 % | vendor-reported | – | – | – | huggingface.co |
Where this comes from
Scores from different settings, harnesses or dates are not directly comparable; the row says which. Sorted by score, highest first, as a reading aid, not a ranking.
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "HumanEval+", https://aurion.semai.id/benchmarks/humaneval-plus/, records of October 2026.