Real-Time Spoken Dialogue and Low-Latency Voice Generation
Models that listen and speak with delays of a few hundred milliseconds, including full-duplex speech-to-speech dialogue and streaming speech generation for voice agents.
Tier Grades
Tier A: Frontier AI labs
feasible
Kyutai built the first real-time full-duplex spoken LLM at 200 ms practical latency with an 8-author team.
Tier B: Indonesian AI labs
stretch
Training a speech-text foundation model like Moshi needs data and compute not stated in the source, so the grade is uncertain.
Tier C: Institution units such as ITB
stretch
Units can study turn-taking and latency with open models such as Moshi but are unlikely to train one.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
API voice models with latency near 200 ms support voice-bot and contact-centre proofs of value.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeMoshi: theoretical latency 160 ms, 200 ms in practice; parameter count and training compute are not in the abstract.
- DataNot in the sources read: the Moshi abstract does not state training hours.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 8 authors on Moshi.
- Time horizonMonths to a year.
- Shelf-life signalsVoice agents reaching human turn-taking latency in Indonesian, Cascaded ASR, LLM and TTS pipelines matching speech-to-speech latency
- Pipeline roleTier A serves real-time voice models; Tier B and Tier C test them on Indonesian conversations; Tier D builds voice agents on them for clients.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
CSM-1Bspeech synthesisexemplar
Moshi (Moshiko)multimodal LLMexemplar
gemini-3.8-livespeech synthesisexemplar
gpt-realtime-2speech synthesisexemplar
Eleven v4 Turbospeech synthesisproduced by
Sonic 3.6speech synthesisproduced byRealtime TTS-2speech synthesisproduced bySonic 3.5speech synthesisproduced byEleven v3 Conversationalspeech synthesisproduced by
Voxtral Mini 4B Realtime 2602speech recognitionproduced byKyutai TTS 1.6B (en, fr)speech synthesisproduced byMimimusic and audioused by
Nova 2 Sonicspeech synthesisproduced by
SeedRealtimespeech synthesisproduced bySonicspeech synthesisproduced byEleven Flash v2.5speech synthesisproduced byEleven Turbo v2.5speech synthesisproduced bygemini-3.8-live-extended-thinkingspeech synthesisproduced bygpt-live-1speech synthesisproduced bygpt-realtime-1.5speech synthesisproduced bygpt-realtime-2.1speech synthesisproduced bygpt-realtime-2.1-minispeech synthesisproduced bygpt-realtime-minispeech synthesisproduced bygpt-realtime-translatespeech synthesisproduced byLinks
R&D TopicReal-Time Spoken Dialogue and Low-Latency Voice Generation
Modelsgemini-3.8-liveGoogle DeepMind, speech synthesisgpt-realtime-2OpenAI, speech synthesisCSM-1BSesame AI Inc., speech synthesisMoshi (Moshiko)Kyutai, multimodal LLM+20 more
gemini-3.8-liveGoogle DeepMind, speech synthesisgpt-realtime-2OpenAI, speech synthesisCSM-1BSesame AI Inc., speech synthesisMoshi (Moshiko)Kyutai, multimodal LLM+20 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 24
SonicCartesia, speech synthesisproduced bySonic 3.5Cartesia, speech synthesisproduced bySonic 3.6Cartesia, speech synthesisproduced byEleven Flash v2.5ElevenLabs, speech synthesisproduced byEleven Turbo v2.5ElevenLabs, speech synthesisproduced byEleven v3 ConversationalElevenLabs, speech synthesisproduced byEleven v4 TurboElevenLabs, speech synthesisproduced bygemini-3.8-liveGoogle DeepMind, speech synthesisexemplargemini-3.8-live-extended-thinkingGoogle DeepMind, speech synthesisproduced bygpt-realtime-translateOpenAI, speech synthesisproduced byMimiKyutai, music and audioused byMoshi (Moshiko)Kyutai, multimodal LLMexemplar+12
Nova 2 SonicAmazon, speech synthesisproduced byVoxtral Mini 4B Realtime 2602Mistral AI, speech recognitionproduced byCSM-1BSesame AI Inc., speech synthesisexemplargpt-live-1OpenAI, speech synthesisproduced bygpt-realtime-1.5OpenAI, speech synthesisproduced bygpt-realtime-2OpenAI, speech synthesisexemplargpt-realtime-2.1OpenAI, speech synthesisproduced bygpt-realtime-2.1-miniOpenAI, speech synthesisproduced bygpt-realtime-miniOpenAI, speech synthesisproduced byKyutai TTS 1.6B (en, fr)Kyutai, speech synthesisproduced bySeedRealtimeByteDance, speech synthesisproduced byRealtime TTS-2speech synthesisproduced byPapers
- Generative Spoken Dialogue Language ModelingTu Anh Nguyen et al., arXiv preprint, 2022 founding
- Qwen2.5-Omni Technical ReportJin Xu et al., arXiv preprint, 2025 state of the art
- Mini-Omni: Language Models Can Hear, Talk While Thinking in StreamingZhifei Xie et al., arXiv preprint, 2024 state of the art
- LLaMA-Omni: Seamless Speech Interaction with Large Language ModelsQingkai Fang et al., ICLR 2025 (per arXiv comment), 2024 state of the art
- Moshi: a speech-text foundation model for real-time dialogueAlexandre Défossez et al., arXiv preprint, 2024 state of the art
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2410.00037 https://arxiv.org/abs/2410.00037
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Real-Time Spoken Dialogue and Low-Latency Voice Generation", https://aurion.semai.id/topics/real-time-voice-interaction/, records of October 2026.