HumanEval
Function-level code generation; saturated legacy metric
codingsaturated
Score in %
Definition
- Version
- not in the notes
- Category
- coding
- Maintainer
- OpenAI
- Status
- saturated
- Home page
- not in the notes
Sources
- ByteDance-Seed/Seed-Coder-8B-Instruct , Hugging Face huggingface.co https://huggingface.co/ByteDance-Seed/Seed-Coder-8B-Instruct
- bigcode/starcoder2-15b , Hugging Face huggingface.co https://huggingface.co/bigcode/starcoder2-15b
Scores at a Glance
Score in %
Higher is better for this unit.
1
Seed-Coder-8B-InstructByteDance
GLM-4 (0520)Z.ai (Zhipu)
Nemotron-4 340BNVIDIAInstruct model; 0-shot
DBRXDatabricksDBRX Instruct; 0-shot pass@1
StarCoder2-15BBigCode (Hugging Face and ServiceNow)
Qwen-14B-ChatAlibaba (Qwen)0-shot pass@1Qwen1.5-72BAlibaba (Qwen)
Seed-Coder-8B-InstructByteDance84.8 %
vendor-reported2GLM-4 (0520)Z.ai (Zhipu)78.5 %
vendor-reported3Nemotron-4 340BNVIDIAInstruct model; 0-shot73.2 %
vendor-reported4DBRXDatabricksDBRX Instruct; 0-shot pass@170.1 %
vendor-reported5StarCoder2-15BBigCode (Hugging Face and ServiceNow)46.3 %
vendor-reported6Qwen-14B-ChatAlibaba (Qwen)0-shot pass@143.9 %
vendor-reported7Qwen1.5-72BAlibaba (Qwen)41.5 %
vendor-reported0255075100
Score in %, Claude 3 model card, Table 1 and vision table
Higher is better for this unit.
1
Claude 3 OpusAnthropic0-shotClaude 3 SonnetAnthropic0-shot
Claude 3 OpusAnthropic0-shot84.9 %
vendor-reported2Claude 3 SonnetAnthropic0-shot73 %
vendor-reported0255075100
Where this comes from
Each bar is one model, with its most independent recorded score. Hatched bars are vendor-reported; the word beside each bar says who ran it. Settings and dates differ between rows; the Scores table below has every result.
Scores
| Model | Score | Who ran it | Setting | View or population | Date | Source |
|---|---|---|---|---|---|---|
| Claude 3 Opus | 84.9 % | vendor-reported | 0-shot | Claude 3 model card, Table 1 and vision table | Mar 2024 | www-cdn.anthropic.com |
| Seed-Coder-8B-Instruct | 84.8 % | vendor-reported | – | – | – | huggingface.co |
 Granite-4.0-H-Micro | 81 % | vendor-reported | pass@1 | Granite-4.0-H-Micro model card | 2 Oct 2025 | huggingface.co |
| GLM-4 (0520) | 78.5 % | vendor-reported | – | – | 18 Jun 2024 | arxiv.org |
| Nemotron-4 340B | 73.2 % | vendor-reported | Instruct model; 0-shot | – | Jun 2024 | huggingface.co |
| Claude 3 Sonnet | 73 % | vendor-reported | 0-shot | Claude 3 model card, Table 1 and vision table | Mar 2024 | www-cdn.anthropic.com |
| DBRX | 70.1 % | vendor-reported | DBRX Instruct; 0-shot pass@1 | – | 27 Mar 2024 | databricks.com |
| StarCoder2-15B | 46.3 % | vendor-reported | – | – | – | huggingface.co |
| Qwen-14B-Chat | 43.9 % | vendor-reported | 0-shot pass@1 | – | Sep 2023 | huggingface.co |
| Qwen1.5-72B | 41.5 % | vendor-reported | – | – | 4 Feb 2024 | qwenlm.github.io |
Where this comes from
Scores from different settings, harnesses or dates are not directly comparable; the row says which. Sorted by score, highest first, as a reading aid, not a ranking.
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "HumanEval", https://aurion.semai.id/benchmarks/humaneval/, records of October 2026.