General Audio and Music Representation Learning
Pre-training audio encoders that classify and describe sound events, music and speech, often without labels.
Tier Grades
Tier A: Frontier AI labs
feasible
Xiaomi trained Dasheng at 1.2 billion parameters on 272,356 hours.
Tier B: Indonesian AI labs
feasible
MERT at 95M to 330M parameters shows useful encoders are within a national lab's reach.
Tier C: Institution units such as ITB
feasible
AST came from a 3-author team and reached 0.485 mAP on AudioSet.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Released classifiers such as YAMNet can tag sounds in an acoustic monitoring proof of concept without training.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeDasheng: 1.2 billion parameters; MERT scales from 95M to 330M parameters.
- DataDasheng: 272,356 hours of diverse audio; AST: 0.485 mAP on AudioSet.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 3 authors on AST; 20 on MERT; 6 on Dasheng.
- Time horizonMonths.
- Shelf-life signalsAudio language models replacing dedicated classifiers
- Pipeline roleTier A and Tier C release encoders; Tier B tunes them for local sounds and music; Tier D uses them for machine sound and media tagging pilots.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
MIT (Jameel Clinic)
Xiaomi
Multimodal Art Projection (m-a-p)
Google
Tencent
Universitat Pompeu Fabra (Music Technology Group)Linked models
Dashengaudio classificationexemplarMERTmusic and audioexemplarAudio Spectrogram Transformer (AST)audio classificationexemplarYAMNetaudio classificationexemplarMuQmusic and audioproduced byMAESTaudio classificationproduced by
CLAP (HTSAT fused)embeddingproduced bySPICEmusic and audioproduced byVGGishaudio classificationproduced byLinks
R&D TopicGeneral Audio and Music Representation Learning
ModelsAudio Spectrogram Transformer (AST)MIT (Jameel Clinic), audio classificationYAMNetGoogle, audio classificationDashengXiaomi, audio classificationMERTMultimodal Art Projection (m-a-p), music and audio+5 more
Audio Spectrogram Transformer (AST)MIT (Jameel Clinic), audio classificationYAMNetGoogle, audio classificationDashengXiaomi, audio classificationMERTMultimodal Art Projection (m-a-p), music and audio+5 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 9
Audio Spectrogram Transformer (AST)MIT (Jameel Clinic), audio classificationexemplarCLAP (HTSAT fused)LAION, embeddingproduced byDashengXiaomi, audio classificationexemplarMAESTUniversitat Pompeu Fabra (Music Technology Group), audio classificationproduced byMERTMultimodal Art Projection (m-a-p), music and audioexemplarMuQTencent, music and audioproduced bySPICEGoogle, music and audioproduced byVGGishGoogle, audio classificationproduced byYAMNetGoogle, audio classificationexemplarPapers
- PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern RecognitionQiuqiang Kong et al., arXiv preprint, 2019 founding
- MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised TrainingYizhi Li et al., ICLR 2024 (per arXiv comment), 2023 state of the art
- CLAP: Learning Audio Concepts From Natural Language SupervisionBenjamin Elizalde et al., arXiv preprint, 2022 state of the art
- Masked Autoencoders that ListenPo-Yao Huang et al., NeurIPS 2022 (per arXiv comment), 2022 state of the art
- BEATs: Audio Pre-Training with Acoustic TokenizersSanyuan Chen et al., arXiv preprint, 2022 state of the art
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2104.01778 https://arxiv.org/abs/2104.01778
- arxiv.org/abs/2406.06992 https://arxiv.org/abs/2406.06992
- arxiv.org/abs/2306.00107 https://arxiv.org/abs/2306.00107
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "General Audio and Music Representation Learning", https://aurion.semai.id/topics/general-audio-music-representations/, records of October 2026.