Video-to-Audio and Foley Generation
Generating sound effects and audio that match what happens in a video, in time and in meaning.
Tier Grades
Tier A: Frontier AI labs
feasible
Tencent built a 100k-hour data pipeline for HunyuanVideo-Foley.
Tier B: Indonesian AI labs
feasible
A 157M parameter model such as MMAudio is small enough for a national lab to retrain.
Tier C: Institution units such as ITB
feasible
MMAudio came from a 6-author academic team at 157M parameters.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
MMAudio generates 8 seconds in 1.23 seconds, fast enough for content production proofs of concept.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeMMAudio has 157M parameters and generates an 8-second clip in 1.23 seconds.
- DataHunyuanVideo-Foley curates a 100k-hour multimodal dataset by automated annotation; MMAudio trains jointly on video and text-audio data.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 6 authors on MMAudio; 8 on HunyuanVideo-Foley.
- Time horizonMonths.
- Shelf-life signalsVideo generators producing synchronised audio natively
- Pipeline roleTier A and Tier C release video-to-audio models; Tier B adds local sound libraries; Tier D applies them in media workflows.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Links
R&D TopicVideo-to-Audio and Foley Generation
Models
HunyuanVideo-FoleyTencent, music and audio
ThinkSoundFunAudioLLM, music and audio
MMAudioUniversity of Illinois Urbana-Champaign, music and audio
FoleyCrafterShanghai AI Lab, music and audio
HunyuanVideo-FoleyTencent, music and audioThinkSoundFunAudioLLM, music and audioMMAudioUniversity of Illinois Urbana-Champaign, music and audioFoleyCrafterShanghai AI Lab, music and audioUse Casesnone recorded
Who Runs Itnone recorded
Papers
- Visually Indicated SoundsAndrew Owens et al., arXiv preprint, 2015 founding
- FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized SoundsYiming Zhang et al., arXiv preprint, 2024 state of the art
- Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow MatchingYongqi Wang et al., NeurIPS 2024 (per arXiv comment), 2024 state of the art
- MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio SynthesisHo Kei Cheng et al., CVPR 2025 (per arXiv comment), 2024 state of the art
- Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion ModelsSimian Luo et al., arXiv preprint, 2023 state of the art
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2412.15322 https://arxiv.org/abs/2412.15322
- arxiv.org/abs/2508.16930 https://arxiv.org/abs/2508.16930
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Video-to-Audio and Foley Generation", https://aurion.semai.id/topics/video-to-audio-foley-generation/, records of October 2026.