Speech-to-Speech Translation
Translating spoken language directly into speech or text in another language, in one model rather than a cascade.
Tier Grades
Tier A: Frontier AI labs
feasible
Meta trained SeamlessM4T with 1 million hours of audio and 68 contributors.
Tier B: Indonesian AI labs
stretch
A national lab could fine-tune SeamlessM4T but not repeat its 1 million hour pre-training.
Tier C: Institution units such as ITB
not feasible
A unit cannot match a 68-person effort on 1 million hours, though it can evaluate the released model.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Released and API speech translation models can run in interpreting pilots.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeNot in the sources read: the SeamlessM4T abstract gives no parameter count or compute.
- DataSeamlessM4T: 1 million hours of open speech audio for w2v-BERT 2.0 pre-training; up to 100 languages.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 68 listed on SeamlessM4T.
- Time horizonMonths to a year.
- Shelf-life signalsReal-time voice models translating speech with natural latency
- Pipeline roleTier A releases speech translation models; Tier B and Tier C evaluate and tune them for Indonesian; Tier D pilots live interpreting.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Links
R&D TopicSpeech-to-Speech Translation
Models
SeamlessM4TMeta, translation
gpt-realtime-translateOpenAI, speech synthesisSeamlessM4T v2 largeMeta, translation
canary-180m-flashNVIDIA, speech recognition+1 more
SeamlessM4TMeta, translationgpt-realtime-translateOpenAI, speech synthesisSeamlessM4T v2 largeMeta, translationcanary-180m-flashNVIDIA, speech recognition+1 moreUse Casesnone recorded
Who Runs Itnone recorded
Papers
- Direct speech-to-speech translation with a sequence-to-sequence modelYe Jia et al., Interspeech 2019 (per arXiv comment), 2019 founding
- SeamlessM4T: Massively Multilingual & Multimodal Machine TranslationSeamless Communication et al., arXiv preprint, 2023 state of the art
- Seamless: Multilingual Expressive and Streaming Speech TranslationSeamless Communication et al., arXiv preprint, 2023 state of the art
- Direct speech-to-speech translation with discrete unitsAnn Lee et al., ACL 2022 (per arXiv comment), 2021 state of the art
- Recent Advances in Direct Speech-to-text TranslationChen Xu et al., IJCAI2023 (per arXiv comment), 2023 survey
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2308.11596 https://arxiv.org/abs/2308.11596
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Speech-to-Speech Translation", https://aurion.semai.id/topics/speech-to-speech-translation/, records of October 2026.