Data Curation and Synthetic Data
Building, filtering and generating training corpora.
Tier Grades
Tier A: Frontier AI labs
feasible
high confidence
Hugging Face and Microsoft are observed building 15T-token and synthetic corpora.
Tier B: Indonesian AI labs
feasible
medium confidence
FineWeb was built by an 8-author team, so corpus building is team-feasible, though processing compute is not sourced.
Tier C: Institution units such as ITB
feasible
medium confidence
Stanford-area groups built Pile of Law (256GB), showing academic-scale curation.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
stretch
low confidence
Domain curation fits client work, but the months-long horizon exceeds a typical proof of concept; the notes give no proof-of-concept anchor.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeProcessing compute not sourced; TabPFN trained purely on synthetic data.
- DataFineWeb 15T tokens from 96 Common Crawl snapshots; FineWeb-Edu 1.3T; Phi-3 synthetic plus filtered web; Pile of Law 256GB legal data with law-grounded filtering.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): FineWeb has 8 authors.
- Time horizonMonths.
- Shelf-life signalsnot in the notes
- Pipeline roleTier B and Tier C build local and domain corpora that feed Tier A's and Tier B's training; Tier D contributes client domain data under proof-of-value agreements.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
Phi-3LLMexemplar
Olmo-Hybrid-7BLLMproduced byOlmo 3.1 32B InstructLLMproduced byOlmo 3LLMproduced byOLMo 2 32B (OLMo-2-0325-32B-Instruct)LLMproduced byOLMo-2-0325-32BLLMproduced by
TabPFNtabularproduced byOlmo 3 7BLLMproduced byOLMo 2LLMproduced by
MAISI (Medical AI for Synthetic Imaging)image generationproduced by
DCLM-7BLLMproduced byNemotron-4 340BLLMproduced by
Falcon 2 11BLLMproduced byOLMo 7B (first release)LLMproduced by
Zephyr 7B betaLLMproduced byFalcon-180BLLMproduced byFalcon-RW-1BLLMproduced byFalcon-40BLLMproduced byFalcon-7B-InstructLLMproduced byFalcon (first generation)LLMproduced byFalcon-7BLLMproduced byOlmo-3-1125-32BLLMproduced byPhi-3.5-miniLLMproduced byLinks
R&D TopicData Curation and Synthetic Data
ModelsPhi-3Microsoft AI, LLMTabPFNPrior Labs, tabularMAISI (Medical AI for Synthetic Imaging)NVIDIA, image generationOLMo 2Allen AI, LLM+19 more
Phi-3Microsoft AI, LLMTabPFNPrior Labs, tabularMAISI (Medical AI for Synthetic Imaging)NVIDIA, image generationOLMo 2Allen AI, LLM+19 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 23
Olmo 3Allen AI, LLMproduced byTabPFNPrior Labs, ELLIS Tübingen, tabularproduced byFalcon (first generation)Technology Innovation Institute (TII), LLMproduced byOlmo 3.1 32B InstructAllen AI, LLMproduced byOlmo-3-1125-32BAllen AI, LLMproduced byOlmo-Hybrid-7BAllen AI, LLMproduced byOLMo 2Allen AI, LLMproduced byOLMo 2 32B (OLMo-2-0325-32B-Instruct)Allen AI, LLMproduced byOlmo 3 7BAllen AI, LLMproduced byOLMo-2-0325-32BAllen AI, LLMproduced byFalcon-7BTechnology Innovation Institute (TII), LLMproduced byFalcon-7B-InstructTechnology Innovation Institute (TII), LLMproduced by+11
Falcon-RW-1BTechnology Innovation Institute (TII), LLMproduced byNemotron-4 340BNVIDIA, LLMproduced byPhi-3Microsoft AI, LLMexemplarFalcon-180BTechnology Innovation Institute (TII), LLMproduced byFalcon-40BTechnology Innovation Institute (TII), LLMproduced byOLMo 7B (first release)Allen AI, LLMproduced byPhi-3.5-miniMicrosoft AI, LLMproduced byZephyr 7B betaHugging Face, LLMproduced byDCLM-7BApple, LLMproduced byFalcon 2 11BTechnology Innovation Institute (TII), LLMproduced byMAISI (Medical AI for Synthetic Imaging)NVIDIA, image generationproduced byPapers
- Textbooks Are All You NeedSuriya Gunasekar et al., arXiv preprint, 2023 founding
- Self-Instruct: Aligning Language Models with Self-Generated InstructionsYizhong Wang et al., ACL 2023 (per arXiv comment), 2022 founding
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at ScaleGuilherme Penedo et al., arXiv preprint, 2024 state of the art
- DataComp-LM: In search of the next generation of training sets for language modelsJeffrey Li et al., arXiv preprint, 2024 state of the art
- Best Practices and Lessons Learned on Synthetic DataRuibo Liu et al., COLM 2024 (per arXiv comment), 2024 survey
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone arxiv.org https://arxiv.org/abs/2404.14219
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale arxiv.org https://arxiv.org/abs/2406.17557
- Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset arxiv.org https://arxiv.org/abs/2207.00220
- Client Challenge nature.com https://www.nature.com/articles/s41586-024-08328-6
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Data Curation and Synthetic Data", https://aurion.semai.id/topics/data-curation-and-synthetic-data/, records of October 2026.