Self-Supervised Speech Representation Learning
Pre-training speech encoders on unlabelled audio so that recognition and other speech tasks need little labelled data.
Tier Grades
Tier A: Frontier AI labs
feasible
Meta and Microsoft pre-trained on 53k to 94k hours and up to 1B parameters.
Tier B: Indonesian AI labs
stretch
Pre-training on tens of thousands of hours of Indonesian audio is a stretch, though fine-tuning the released encoders is not.
Tier C: Institution units such as ITB
feasible
wav2vec 2.0 shows 10 minutes of labelled data can give usable recognition, which suits a unit with small labelled sets.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Released encoders such as XLSR-53 already underlie an Indonesian, Javanese and Sundanese recogniser, so they can be fine-tuned for a proof of concept.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeHuBERT scales to a 1B parameter model; wav2vec 2.0 and WavLM abstracts give no training compute.
- Datawav2vec 2.0: 53k hours unlabelled, then 10 minutes labelled gives 4.8/8.2 WER; WavLM: training data scaled from 60k to 94k hours.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 4 authors on wav2vec 2.0; 19 on WavLM.
- Time horizonMonths to fine-tune; a year or more to pre-train at 94k hours.
- Shelf-life signalsSupervised or audio language models outperforming self-supervised encoders on low-resource languages
- Pipeline roleTier A pre-trains encoders on unlabelled audio; Tier C and Tier B fine-tune them with small labelled sets; Tier D uses the fine-tuned encoders in speech pilots.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
w2v-bert-2.0speech recognitionexemplar
WavLMspeech recognitionexemplarHuBERTspeech recognitionexemplarwav2vec 2.0speech recognitionexemplarwavlm-base-plusspeech recognitionproduced bywavlm-largespeech recognitionproduced byhubert-base-ls960speech recognitionproduced byhubert-large-ll60kspeech recognitionproduced byhubert-xlarge-ll60kspeech recognitionproduced bywav2vec2-large-xlsr-53speech recognitionproduced by
TRILLssonembeddingproduced byLinks
R&D TopicSelf-Supervised Speech Representation Learning
Modelsw2v-bert-2.0Meta, speech recognitionwav2vec 2.0Meta, speech recognitionHuBERTMeta, speech recognitionWavLMMicrosoft AI, speech recognition+7 more
w2v-bert-2.0Meta, speech recognitionwav2vec 2.0Meta, speech recognitionHuBERTMeta, speech recognitionWavLMMicrosoft AI, speech recognition+7 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 11
HuBERTMeta, speech recognitionexemplarWavLMMicrosoft AI, speech recognitionexemplarhubert-base-ls960Meta, speech recognitionproduced byhubert-large-ll60kMeta, speech recognitionproduced byhubert-xlarge-ll60kMeta, speech recognitionproduced byTRILLssonGoogle, embeddingproduced bywav2vec 2.0Meta, speech recognitionexemplarwav2vec2-large-xlsr-53Meta, speech recognitionproduced bywavlm-base-plusMicrosoft AI, speech recognitionproduced bywavlm-largeMicrosoft AI, speech recognitionproduced byw2v-bert-2.0Meta, speech recognitionexemplarPapers
- wav2vec: Unsupervised Pre-training for Speech RecognitionSteffen Schneider et al., arXiv preprint, 2019 founding
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski et al., arXiv preprint, 2020 founding
- data2vec: A General Framework for Self-supervised Learning in Speech, Vision and LanguageAlexei Baevski et al., arXiv preprint, 2022 state of the art
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsWei-Ning Hsu et al., arXiv preprint, 2021 state of the art
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech ProcessingSanyuan Chen et al., arXiv preprint, 2021 state of the art
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2006.11477 https://arxiv.org/abs/2006.11477
- arxiv.org/abs/2106.07447 https://arxiv.org/abs/2106.07447
- arxiv.org/abs/2110.13900 https://arxiv.org/abs/2110.13900
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Self-Supervised Speech Representation Learning", https://aurion.semai.id/topics/self-supervised-speech-representations/, records of October 2026.