Audio-Driven Talking Heads and Lip-Sync
Generating a talking person video from one photo and an audio track, or re-syncing lips to new audio for dubbing.
Tier Grades
Tier A: Frontier AI labs
feasible
Big tech trains human video models on large data.
Tier B: Indonesian AI labs
stretch
Training is a stretch for an Indonesian lab; open models such as LatentSync can be fine-tuned.
Tier C: Institution units such as ITB
feasible
University units publish talking-head methods such as SadTalker.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Avatars and dubbing in Bahasa Indonesia are clear pilots, with consent checks.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeTraining compute is not in the sources read.
- DataNot in the sources read; OmniHuman-1 mixes motion-related conditions into training to scale data.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 5 authors on the OmniHuman-1 paper (arXiv 2502.01061).
- Time horizonMonths.
- Shelf-life signalsReal-time interactive avatars, Deepfake fraud drives consent and watermark rules
- Pipeline roleTier A trains avatar models; Tier C adapts open ones; Tier D builds dubbing and avatar pilots with consent controls.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
Audio2Face-3Davatarexemplar
OmniHuman-1avatarexemplarLatentSync 1.6avatarexemplar
SadTalkeravatarexemplar
Synthesia 3.0avatarproduced byOmniHuman-1.5avatarproduced by
Wan2.2-S2V-14Bvideo generationproduced by
lipsync-2avatarproduced byAudio2Emotionavatarproduced by
MultiTalkavatarproduced by
HeyGen Avatar IVavatarproduced byOmniHumanavatarproduced byreact-1avatarproduced by
VEED Fabric 1.0avatarproduced byLinks
R&D TopicAudio-Driven Talking Heads and Lip-Sync
ModelsLatentSync 1.6ByteDance, avatarAudio2Face-3DNVIDIA, avatarOmniHuman-1ByteDance, avatarSadTalkerXi'an Jiaotong University, avatar+10 more
LatentSync 1.6ByteDance, avatarAudio2Face-3DNVIDIA, avatarOmniHuman-1ByteDance, avatarSadTalkerXi'an Jiaotong University, avatar+10 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 14
OmniHumanByteDance, avatarproduced byHeyGen Avatar IVHeyGen, avatarproduced byOmniHuman-1ByteDance, avatarexemplarOmniHuman-1.5ByteDance, avatarproduced bySadTalkerXi'an Jiaotong University, Tencent, avatarexemplarWan2.2-S2V-14BAlibaba (Qwen), video generationproduced byAudio2EmotionNVIDIA, avatarproduced byAudio2Face-3DNVIDIA, avatarexemplarLatentSync 1.6ByteDance, avatarexemplarlipsync-2Sync Labs, avatarproduced byMultiTalkMeituan, avatarproduced byreact-1Sync Labs, avatarproduced byPapers
- A Lip Sync Expert Is All You Need for Speech to Lip Generation In The WildK R Prajwal et al., arXiv preprint, 2020 founding
- EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak ConditionsLinrui Tian et al., arXiv preprint, 2024 state of the art
- Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image AnimationMingwang Xu et al., arXiv preprint, 2024 state of the art
- LivePortrait: Efficient Portrait Animation with Stitching and Retargeting ControlJianzhu Guo et al., arXiv preprint, 2024 state of the art
- SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face AnimationWenxuan Zhang et al., CVPR 2023 (per arXiv comment), 2022 state of the art
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2502.01061 https://arxiv.org/abs/2502.01061
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Audio-Driven Talking Heads and Lip-Sync", https://aurion.semai.id/topics/audio-driven-talking-heads-and-lip-sync/, records of October 2026.