Neural Audio Codecs and Audio Tokenisers
Learned encoders and decoders that compress audio into discrete tokens, used for transmission and as the vocabulary of speech and music generators.
Tier Grades
Tier A: Frontier AI labs
feasible
Google and Kyutai built the codecs behind their speech and music generators.
Tier B: Indonesian AI labs
feasible
A codec that runs in real time on a smartphone CPU is small enough to train at a national lab.
Tier C: Institution units such as ITB
feasible
SoundStream came from a 5-author team, a size a unit can match.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
stretch
Delivery work uses codecs inside released generators rather than building them.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeSoundStream runs in real time on a smartphone CPU and covers 3 to 18 kbps in one model.
- DataNot in the sources read: the SoundStream abstract does not state training data size.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 5 authors on SoundStream.
- Time horizonMonths.
- Shelf-life signalsGenerators that work on continuous latents instead of discrete codec tokens
- Pipeline roleTier A and Tier C build codecs; Tier B uses them as tokenisers for Indonesian speech models; Tier D receives them inside TTS and dialogue systems.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
Links
R&D TopicNeural Audio Codecs and Audio Tokenisers
Use Casesnone recorded
Who Runs Itnone recorded
Papers
- SoundStream: An End-to-End Neural Audio CodecNeil Zeghidour et al., arXiv preprint, 2021 founding
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language ModelingShengpeng Ji et al., ICLR 2025 (per arXiv comment), 2024 state of the art
- High-Fidelity Audio Compression with Improved RVQGANRithesh Kumar et al., NeurIPS 2023 (per arXiv comment), 2023 state of the art
- SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language ModelsXin Zhang et al., ICLR 2024 (per arXiv comment), 2023 state of the art
- Codec-SUPERB: An In-Depth Analysis of Sound Codec ModelsHaibin Wu et al., arXiv preprint, 2024 survey
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2107.03312 https://arxiv.org/abs/2107.03312
- arxiv.org/abs/2410.00037 https://arxiv.org/abs/2410.00037
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Neural Audio Codecs and Audio Tokenisers", https://aurion.semai.id/topics/neural-audio-codecs/, records of October 2026.