Expressive and Controllable Speech Synthesis
Controlling the style, emotion and identity of synthetic speech from a text description or a sampled style rather than only a reference recording.
Tier Grades
Tier A: Frontier AI labs
feasible
Frontier labs can label and train at the 45k hour scale used for natural-language guidance.
Tier B: Indonesian AI labs
feasible
The 45k hour labelling method uses found data, which an Indonesian lab can apply to local speech.
Tier C: Institution units such as ITB
feasible
StyleTTS 2 reached human-level naturalness on public sets with a 5-author team.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Released description-controlled models can voice branded or emotional content in a proof of concept.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeNot in the sources read: StyleTTS 2 and the Parler-TTS paper abstracts give no training compute.
- DataParler-TTS paper: automatic labels of speaker identity and style applied to a 45k hour dataset; StyleTTS 2: LJSpeech, VCTK and LibriTTS.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 2 authors on the natural-language guidance paper; 5 on StyleTTS 2.
- Time horizonMonths.
- Shelf-life signalsGeneral TTS models accepting free-text style prompts as standard
- Pipeline roleTier A and Tier C publish control methods; Tier B adapts them to Indonesian prosody; Tier D applies them to content and customer voice pilots.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
Links
R&D TopicExpressive and Controllable Speech Synthesis
Models
StyleTTS 2Columbia University, speech synthesis
Maya1Maya Research, speech synthesis
Vevo2Amphion (open-mmlab/Amphion), speech synthesis
FlowtronNVIDIA, speech synthesis+3 more
StyleTTS 2Columbia University, speech synthesisMaya1Maya Research, speech synthesisVevo2Amphion (open-mmlab/Amphion), speech synthesisFlowtronNVIDIA, speech synthesis+3 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 7
ai4bharat indic-parler-ttsAI4Bharat (IIT Madras), speech synthesisproduced byFastPitchNVIDIA, speech synthesisproduced by
Nari Dia-1.6BNari Labs, speech synthesisproduced byFlowtronNVIDIA, speech synthesisproduced byMaya1Maya Research, speech synthesisexemplarStyleTTS 2Columbia University, speech synthesisexemplarVevo2Amphion (open-mmlab/Amphion), speech synthesisexemplarPapers
- Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech SynthesisYuxuan Wang et al., arXiv preprint, 2018 founding
- Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with TacotronRJ Skerry-Ryan et al., arXiv preprint, 2018 founding
- Natural language guidance of high-fidelity text-to-speech with synthetic annotationsDan Lyth et al., arXiv preprint, 2024 state of the art
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing SynthesizersKai Shen et al., arXiv preprint, 2023 state of the art
- StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language ModelsYinghao Aaron Li et al., NeurIPS 2023 (per arXiv comment), 2023 state of the art
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2306.07691 https://arxiv.org/abs/2306.07691
- arxiv.org/abs/2402.01912 https://arxiv.org/abs/2402.01912
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Expressive and Controllable Speech Synthesis", https://aurion.semai.id/topics/expressive-controllable-speech-synthesis/, records of October 2026.