Evaluation and Benchmark Building
Designing test sets that measure capability or risk.
Tier Grades
Tier A: Frontier AI labs
feasible
medium confidence
Frontier labs are observed (SWE-bench Verified built with OpenAI).
Tier B: Indonesian AI labs
feasible
medium confidence
Benchmarks are built with human effort and API inference, which b can fund without training compute.
Tier C: Institution units such as ITB
feasible
medium confidence
GAIA was built by 6 authors, an academic-scale team.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
low confidence
Small domain test sets need human effort and API calls only, fitting proof-of-value measurement, though the notes give no direct proof-of-concept anchor.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeMainly expert human effort and API inference, not training compute (inferred in the notes; no cost figures).
- DataHLE 2,500 questions; GAIA 466 questions; SWE-bench Verified 500 human-filtered instances.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): HLE about 1,000 authors; GAIA 6 authors.
- Time horizonMonths to build; months to saturate.
- Shelf-life signalsHLE built because LLMs exceed 90% on MMLU, Agent benchmarks moving fast, ARC-AGI-1 saturation leading to ARC-AGI-2 (unverified, page fetch failed)
- Pipeline roleTier C and Tier B build benchmarks (including local-language ones) that grade Tier A's releases; Tier D uses evaluation to show proof of value to clients.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
NonprofitsFrontier labsAcademia
Linked models
Links
R&D TopicEvaluation and Benchmark Building
Use Casesnone recorded
Who Runs Itnone recorded
Papers
- Measuring Massive Multitask Language UnderstandingDan Hendrycks et al., ICLR 2021 (per arXiv comment), 2020 founding
- Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelsAarohi Srivastava et al., Transactions on Machine Learning Research, May/2022, https://openreview.net/foru, 2022 founding
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human PreferenceWei-Lin Chiang et al., arXiv preprint, 2024 state of the art
- Holistic Evaluation of Language ModelsPercy Liang et al., Published in Transactions on Machine Learning Research (TMLR), 2023, 2022 state of the art
- Evaluating Large Language Models: A Comprehensive SurveyZishan Guo et al., arXiv preprint, 2023 survey
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- GAIA: a benchmark for General AI Assistants arxiv.org https://arxiv.org/abs/2311.12983
- SWE-bench Leaderboards swebench.com https://www.swebench.com/
- Humanity's Last Exam arxiv.org https://arxiv.org/abs/2501.14249
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Evaluation and Benchmark Building", https://aurion.semai.id/topics/evaluation-and-benchmark-building/, records of October 2026.