Video Editing and Controllable Video Generation
Editing existing video, or steering generation with references, masks, poses or motion modules.
Tier Grades
Tier A: Frontier AI labs
feasible
Big labs release editing models built on their video bases.
Tier B: Indonesian AI labs
stretch
Fine-tuning editing models needs a video base model, a stretch for an Indonesian lab.
Tier C: Institution units such as ITB
feasible
A 6-author team produced VACE; a university unit can work on adapters.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Video localisation and object removal are clear media pilots.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeTraining compute is not in the sources read.
- DataNot in the sources read; VACE unifies reference-to-video, video-to-video and masked editing.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 6 authors on the VACE paper (arXiv 2503.07598).
- Time horizonMonths.
- Shelf-life signalsGeneral video models accept edit instructions, Real-time video editing reaches consumer devices
- Pipeline roleTier A releases editing models; Tier C builds adapters; Tier D uses them in media pilots.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
VOIDvideo generationexemplar
Lucy Edit Devvideo generationexemplar
Wan2.1-VACE (1.3B, 14B)video generationexemplar
AnimateDiffvideo generationexemplarWan-Fun (Wan2.1-Fun, Wan2.2-Fun)video generationproduced byLive2Diffvideo generationproduced by
AnimateDiff-Lightningvideo generationproduced by
AnimateLCMvideo generationproduced byPIA (Personalized Image Animator)video generationproduced byAnimateDiffvideo generationproduced by
vid2vidvideo generationproduced byLucy 2.0video generationproduced byLucy Edit 2video generationproduced byLinks
R&D TopicVideo Editing and Controllable Video Generation
ModelsWan2.1-VACE (1.3B, 14B)Alibaba (Qwen), video generationAnimateDiffShanghai AI Lab, video generationLucy Edit DevDecart, video generationVOIDNetflix, video generation+9 more
Wan2.1-VACE (1.3B, 14B)Alibaba (Qwen), video generationAnimateDiffShanghai AI Lab, video generationLucy Edit DevDecart, video generationVOIDNetflix, video generation+9 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 13
AnimateDiffShanghai AI Lab, The Chinese University of Hong Kong, Stanford University, video generationproduced byAnimateDiffShanghai AI Lab, The Chinese University of Hong Kong, Stanford University, video generationexemplarAnimateLCMThe Chinese University of Hong Kong, SenseTime, video generationproduced byAnimateDiff-LightningByteDance, video generationproduced byLucy 2.0Decart, video generationproduced byvid2vidNVIDIA, video generationproduced byWan-Fun (Wan2.1-Fun, Wan2.2-Fun)Alibaba (Qwen), video generationproduced byWan2.1-VACE (1.3B, 14B)Alibaba (Qwen), video generationexemplarLive2DiffShanghai AI Lab, video generationproduced byLucy Edit 2Decart, video generationproduced byLucy Edit DevDecart, video generationexemplarPIA (Personalized Image Animator)Shanghai AI Lab, video generationproduced byPapers
- Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video GenerationJay Zhangjie Wu et al., arXiv preprint, 2022 founding
- VACE: All-in-One Video Creation and EditingZeyinzi Jiang et al., arXiv preprint, 2025 state of the art
- Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video GeneratorsLevon Khachatryan et al., arXiv preprint, 2023 state of the art
- AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific TuningYuwei Guo et al., arXiv preprint, 2023 state of the art
- A Survey on Video Diffusion ModelsZhen Xing et al., arXiv preprint, 2023 survey
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2503.07598 https://arxiv.org/abs/2503.07598
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Video Editing and Controllable Video Generation", https://aurion.semai.id/topics/video-editing-and-controllable-video-generation/, records of October 2026.