Large-Scale Zero-Shot Text-to-Speech
Text-to-speech trained on tens of thousands of hours that speaks in an unseen voice from a few seconds of reference audio.
Tier Grades
Tier A: Frontier AI labs
feasible
Microsoft scaled TTS training to 60K hours for VALL-E.
Tier B: Indonesian AI labs
feasible
F5-TTS shows a public 100K hours dataset is enough, which an Indonesian lab can combine with local speech.
Tier C: Institution units such as ITB
stretch
Collecting tens of thousands of hours is a stretch for a unit, though fine-tuning released models such as F5-TTS is not.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Released models clone a voice from 3 seconds of audio, enough for a proof of concept with consented voices.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeF5-TTS reaches an inference real-time factor of 0.15; VALL-E, F5-TTS and CosyVoice abstracts give no training compute.
- DataVALL-E: 60K hours of English speech and a 3-second prompt; F5-TTS: a public 100K hours multilingual dataset.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 13 authors on VALL-E; 8 on F5-TTS.
- Time horizonMonths.
- Shelf-life signalsVoice cloning restricted by law or platform policy, Open models matching commercial arena leaders
- Pipeline roleTier A and open labs release zero-shot TTS; Tier B adds Indonesian data; Tier C evaluates naturalness; Tier D builds voice products with consented voices.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Microsoft AI
FunAudioLLMSWivid
ElevenLabs
MiniMax
Fish Audio
Amphion (open-mmlab/Amphion)
Coqui
Resemble AI
ZyphraLinked models
F5-TTSspeech synthesisexemplarXTTS-v2speech synthesisexemplarVALL-Espeech synthesisexemplarCosyVoice 2speech synthesisexemplar
Qwen-Audio-3.1-TTS-Plusspeech synthesisproduced byEleven v4speech synthesisproduced by
Gemini 3.8 Flash-Lite TTSspeech synthesisproduced byGemini 3.8 Flash TTSspeech synthesisproduced by
Breeze TTS 2speech synthesisproduced byStepAudio 2.5 TTSspeech synthesisproduced byHiggs-Audio-v3-Studiospeech synthesisproduced bySimba 3.2speech synthesisproduced by
MOSS-TTS v1.5speech synthesisproduced byGemini 3.1 Flash TTSspeech synthesisproduced by
OmniVoicespeech synthesisproduced byFish Audio S2 Prospeech synthesisproduced by
SpaceXAI TTSspeech synthesisproduced bySpeech 2.8 HDspeech synthesisproduced byQwen3-TTS 12Hz 0.6Bspeech synthesisproduced byQwen3-TTS 12Hz 1.7Bspeech synthesisproduced byFun-CosyVoice3-0.5B-2512speech synthesisproduced by
NeuTTS Airspeech synthesisproduced byMAI-Voice-1speech synthesisproduced byEleven v3speech synthesisproduced by
XTTS-v2speech synthesisexemplarVALL-Espeech synthesisexemplarCosyVoice 2speech synthesisexemplarQwen-Audio-3.1-TTS-Plusspeech synthesisproduced byEleven v4speech synthesisproduced byGemini 3.8 Flash-Lite TTSspeech synthesisproduced byGemini 3.8 Flash TTSspeech synthesisproduced byBreeze TTS 2speech synthesisproduced byStepAudio 2.5 TTSspeech synthesisproduced byHiggs-Audio-v3-Studiospeech synthesisproduced bySimba 3.2speech synthesisproduced byMOSS-TTS v1.5speech synthesisproduced byGemini 3.1 Flash TTSspeech synthesisproduced byOmniVoicespeech synthesisproduced byFish Audio S2 Prospeech synthesisproduced bySpaceXAI TTSspeech synthesisproduced bySpeech 2.8 HDspeech synthesisproduced byQwen3-TTS 12Hz 0.6Bspeech synthesisproduced byQwen3-TTS 12Hz 1.7Bspeech synthesisproduced byFun-CosyVoice3-0.5B-2512speech synthesisproduced byNeuTTS Airspeech synthesisproduced byMAI-Voice-1speech synthesisproduced byEleven v3speech synthesisproduced by+29
Orpheus TTSspeech synthesisproduced byVevospeech synthesisproduced byZonos v0.1speech synthesisproduced byMetisspeech synthesisproduced byMaskGCTspeech synthesisproduced by
E2 TTSspeech synthesisproduced byChatTTSspeech synthesisproduced by
OpenVoicespeech synthesisproduced byTortoise TTSspeech synthesisproduced by
Barkspeech synthesisproduced byYourTTSspeech synthesisproduced bySpeechT5speech synthesisproduced byQwen TTS (Model Studio)speech synthesisproduced byBoson higgs-tts-3-4bspeech synthesisproduced byElevenspeech synthesisproduced byFish Speechspeech synthesisproduced byCosyVoice 1speech synthesisproduced byIndexTTS-2.5speech synthesisproduced byMAI-Voice-2.1speech synthesisproduced byVibeVoice-1.5Bspeech synthesisproduced bySpeech 2.8 Turbospeech synthesisproduced by
gpt-4o-mini-ttsspeech synthesisproduced by
OpenBMB VoxCPM2speech synthesisproduced by
Pika Speechspeech synthesisproduced byResemble Chatterboxspeech synthesisproduced byGrok Voice APIspeech synthesisproduced by
AuKspeech synthesisproduced by
Nari Dia-1.6Bspeech synthesisproduced bySupertone supertonic-3speech synthesisproduced byLinks
R&D TopicLarge-Scale Zero-Shot Text-to-Speech
ModelsXTTS-v2Coqui, speech synthesisCosyVoice 2FunAudioLLM, speech synthesisF5-TTSSWivid, speech synthesisVALL-EMicrosoft AI, speech synthesis+49 more
XTTS-v2Coqui, speech synthesisCosyVoice 2FunAudioLLM, speech synthesisF5-TTSSWivid, speech synthesisVALL-EMicrosoft AI, speech synthesis+49 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 53
ElevenElevenLabs, speech synthesisproduced byEleven v4ElevenLabs, speech synthesisproduced byEleven v3ElevenLabs, speech synthesisproduced byCosyVoice 2FunAudioLLM, speech synthesisexemplarCosyVoice 1FunAudioLLM, speech synthesisproduced byE2 TTSMicrosoft, speech synthesisproduced byF5-TTSSWivid, speech synthesisexemplarFish Audio S2 ProFish Audio, speech synthesisproduced byFish SpeechFish Audio, speech synthesisproduced byFun-CosyVoice3-0.5B-2512FunAudioLLM, speech synthesisproduced byGemini 3.1 Flash TTSGoogle, speech synthesisproduced byGemini 3.8 Flash TTSGoogle, speech synthesisproduced by+41
Higgs-Audio-v3-Studiodrbaph, speech synthesisproduced byNari Dia-1.6BNari Labs, speech synthesisproduced byNeuTTS AirNeuphonic, speech synthesisproduced byOmniVoicek2-fsa, speech synthesisproduced byOpenVoiceMyShell.ai, MIT (Jameel Clinic), speech synthesisproduced byQwen-Audio-3.1-TTS-PlusAlibaba (Qwen), speech synthesisproduced byQwen3-TTS 12Hz 1.7BAlibaba (Qwen), speech synthesisproduced byAuKTencent, speech synthesisproduced byBarkSuno, Inc., speech synthesisproduced byBoson higgs-tts-3-4bspeech synthesisproduced byBreeze TTS 2BreezeBlue, speech synthesisproduced byChatTTS2Noise, speech synthesisproduced byGemini 3.8 Flash-Lite TTSGoogle, speech synthesisproduced bygpt-4o-mini-ttsOpenAI, speech synthesisproduced byGrok Voice APISpaceXAI (xAI), speech synthesisproduced byMAI-Voice-1Microsoft AI, speech synthesisproduced byMAI-Voice-2.1Microsoft AI, speech synthesisproduced byMaskGCTAmphion (open-mmlab/Amphion), speech synthesisproduced byMetisAmphion (open-mmlab/Amphion), speech synthesisproduced byMOSS-TTS v1.5OpenMOSS, speech synthesisproduced byOpenBMB VoxCPM2OpenBMB (Tsinghua NLP Lab and ModelBest), speech synthesisproduced byOrpheus TTSCanopy Labs, speech synthesisproduced byPika SpeechPika (Pika Labs), speech synthesisproduced byQwen TTS (Model Studio)Alibaba (Qwen), speech synthesisproduced byQwen3-TTS 12Hz 0.6BAlibaba (Qwen), speech synthesisproduced byResemble ChatterboxResemble AI, speech synthesisproduced bySpaceXAI TTSSpaceXAI (xAI), speech synthesisproduced bySpeech 2.8 HDMiniMax, speech synthesisproduced bySpeech 2.8 TurboMiniMax, speech synthesisproduced bySpeechT5Microsoft AI, speech synthesisproduced bySupertone supertonic-3speech synthesisproduced byTortoise TTSJames Betker, speech synthesisproduced byVALL-EMicrosoft AI, speech synthesisexemplarVevoAmphion (open-mmlab/Amphion), speech synthesisproduced byVibeVoice-1.5BMicrosoft AI, speech synthesisproduced byXTTS-v2Coqui, speech synthesisexemplarYourTTSCoqui, speech synthesisproduced byZonos v0.1Zyphra, speech synthesisproduced byIndexTTS-2.5speech synthesisproduced bySimba 3.2speech synthesisproduced byStepAudio 2.5 TTSspeech synthesisproduced by
Nari Dia-1.6BNari Labs, speech synthesisproduced byNeuTTS AirNeuphonic, speech synthesisproduced byOmniVoicek2-fsa, speech synthesisproduced byOpenVoiceMyShell.ai, MIT (Jameel Clinic), speech synthesisproduced byQwen-Audio-3.1-TTS-PlusAlibaba (Qwen), speech synthesisproduced byQwen3-TTS 12Hz 1.7BAlibaba (Qwen), speech synthesisproduced byAuKTencent, speech synthesisproduced byBarkSuno, Inc., speech synthesisproduced byBoson higgs-tts-3-4bspeech synthesisproduced byBreeze TTS 2BreezeBlue, speech synthesisproduced byChatTTS2Noise, speech synthesisproduced byGemini 3.8 Flash-Lite TTSGoogle, speech synthesisproduced bygpt-4o-mini-ttsOpenAI, speech synthesisproduced byGrok Voice APISpaceXAI (xAI), speech synthesisproduced byMAI-Voice-1Microsoft AI, speech synthesisproduced byMAI-Voice-2.1Microsoft AI, speech synthesisproduced byMaskGCTAmphion (open-mmlab/Amphion), speech synthesisproduced byMetisAmphion (open-mmlab/Amphion), speech synthesisproduced byMOSS-TTS v1.5OpenMOSS, speech synthesisproduced byOpenBMB VoxCPM2OpenBMB (Tsinghua NLP Lab and ModelBest), speech synthesisproduced byOrpheus TTSCanopy Labs, speech synthesisproduced byPika SpeechPika (Pika Labs), speech synthesisproduced byQwen TTS (Model Studio)Alibaba (Qwen), speech synthesisproduced byQwen3-TTS 12Hz 0.6BAlibaba (Qwen), speech synthesisproduced byResemble ChatterboxResemble AI, speech synthesisproduced bySpaceXAI TTSSpaceXAI (xAI), speech synthesisproduced bySpeech 2.8 HDMiniMax, speech synthesisproduced bySpeech 2.8 TurboMiniMax, speech synthesisproduced bySpeechT5Microsoft AI, speech synthesisproduced bySupertone supertonic-3speech synthesisproduced byTortoise TTSJames Betker, speech synthesisproduced byVALL-EMicrosoft AI, speech synthesisexemplarVevoAmphion (open-mmlab/Amphion), speech synthesisproduced byVibeVoice-1.5BMicrosoft AI, speech synthesisproduced byXTTS-v2Coqui, speech synthesisexemplarYourTTSCoqui, speech synthesisproduced byZonos v0.1Zyphra, speech synthesisproduced byIndexTTS-2.5speech synthesisproduced bySimba 3.2speech synthesisproduced byStepAudio 2.5 TTSspeech synthesisproduced byPapers
- Neural Codec Language Models are Zero-Shot Text to Speech SynthesizersChengyi Wang et al., arXiv preprint, 2023 founding
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion ModelsZeqian Ju et al., arXiv preprint, 2024 state of the art
- XTTS: a Massively Multilingual Zero-Shot Text-to-Speech ModelEdresson Casanova et al., INTERSPEECH 2024 (per arXiv comment), 2024 state of the art
- F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow MatchingYushen Chen et al., arXiv preprint, 2024 state of the art
- CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic TokensZhihao Du et al., arXiv preprint, 2024 state of the art
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2301.02111 https://arxiv.org/abs/2301.02111
- arxiv.org/abs/2410.06885 https://arxiv.org/abs/2410.06885
- arxiv.org/abs/2407.05407 https://arxiv.org/abs/2407.05407
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Large-Scale Zero-Shot Text-to-Speech", https://aurion.semai.id/topics/zero-shot-voice-cloning-tts/, records of October 2026.