Video Understanding with Language Models
Vision-language models that summarise, search and answer questions about long videos.
Tier Grades
Tier A: Frontier AI labs
feasible
Alibaba, Meta and Shanghai AI Lab release video LLMs.
Tier B: Indonesian AI labs
stretch
Video training data and compute are a stretch; Indonesian video LLM work is not in the sources.
Tier C: Institution units such as ITB
feasible
Fine-tuning a released 7B video LLM (VideoLLaMA3) on local footage is institution-scale.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
CCTV review, broadcast archive search and site monitoring proofs of concept can use released 7B models.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeVideoLLaMA3 is trained in four stages from vision encoder adaptation to video fine-tuning; compute is not in the abstract.
- DataVideoLLaMA3 favours large high-quality image-text data (scenes, documents, charts) over massive video-text data.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 15 authors on VideoLLaMA3.
- Time horizonMonths.
- Shelf-life signalsToken compression for hour-long video (LongVU), Video search products built on video embeddings and video LLMs (TwelveLabs Marengo and Pegasus)
- Pipeline roleTier A releases video LLMs; Tier C adapts them to local video; Tier D applies them to monitoring and archive search.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
VideoLLaMA3 7Bmultimodal LLMexemplar
LongVUmultimodal LLMexemplar
TwelveLabs Pegasusmultimodal LLMexemplar
Molmo 2multimodal LLMproduced byMolmo2-8Bmultimodal LLMproduced by
Keye-VL 1.5 8Bmultimodal LLMproduced byKeye-VLmultimodal LLMproduced byKeye-VL 8B Previewmultimodal LLMproduced by
SmolVLM2 256M Video Instructmultimodal LLMproduced bySmolVLM2 500M Video Instructmultimodal LLMproduced by
InternVideo2.5 Chat 8Bmultimodal LLMproduced byVideoLLaMA2.1 7B 16Fmultimodal LLMproduced by
LLaVA-NeXT-Video 7Bmultimodal LLMproduced byVideoLLaMAmultimodal LLMproduced by
AMIEmultimodal LLMproduced by
video-SALMONN 2+ 72Bmultimodal LLMproduced byTwelveLabs Marengoembeddingproduced byMarengo 3.0embeddingproduced byPegasus 1.5multimodal LLMproduced byLinks
R&D TopicVideo Understanding with Language Models
Models 19
AMIEGoogle, Google DeepMind, multimodal LLMproduced byKeye-VLKuaishou, multimodal LLMproduced byLLaVA-NeXT-Video 7BUniversity of Wisconsin-Madison, Microsoft AI, multimodal LLMproduced byMolmo2-8BAllen AI, multimodal LLMproduced bySmolVLM2 256M Video InstructHugging Face, multimodal LLMproduced bySmolVLM2 500M Video InstructHugging Face, multimodal LLMproduced byKeye-VL 1.5 8BKuaishou, multimodal LLMproduced byKeye-VL 8B PreviewKuaishou, multimodal LLMproduced byMarengo 3.0TwelveLabs, embeddingproduced byMolmo 2Allen AI, multimodal LLMproduced byTwelveLabs MarengoTwelveLabs, embeddingproduced byVideoLLaMAAlibaba DAMO, multimodal LLMproduced by+7
InternVideo2.5 Chat 8BShanghai AI Lab, multimodal LLMproduced byLongVUMeta, King Abdullah University of Science and Technology, multimodal LLMexemplarPegasus 1.5TwelveLabs, multimodal LLMproduced byTwelveLabs PegasusTwelveLabs, multimodal LLMexemplarvideo-SALMONN 2+ 72BTsinghua University, ByteDance, multimodal LLMproduced byVideoLLaMA2.1 7B 16FAlibaba DAMO, multimodal LLMproduced byVideoLLaMA3 7BAlibaba DAMO, multimodal LLMexemplarPapers
- VideoChat: Chat-Centric Video UnderstandingKunChang Li et al., arXiv preprint, 2023 founding
- VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video UnderstandingBoqiang Zhang et al., arXiv preprint, 2025 state of the art
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video AnalysisChaoyou Fu et al., arXiv preprint, 2024 state of the art
- LLaVA-Video: Video Instruction Tuning With Synthetic DataYuanhan Zhang et al., arXiv preprint, 2024 state of the art
- Video Understanding with Large Language Models: A SurveyYolo Y. Tang et al., arXiv preprint, 2023 survey
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2501.13106 https://arxiv.org/abs/2501.13106
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Video Understanding with Language Models", https://aurion.semai.id/topics/video-understanding-with-language-models/, records of October 2026.