Video Action Recognition and Temporal Understanding
Classifying and locating human actions and events in video clips.
Tier Grades
Tier A: Frontier AI labs
feasible
Billion-parameter video pretraining is frontier and big-tech scale.
Tier B: Indonesian AI labs
stretch
Pretraining at that scale is a stretch; fine-tuning released video models on local footage is feasible.
Tier C: Institution units such as ITB
feasible
Benchmarking on Kinetics and Something-Something fits a university unit.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Released action models support safety and process monitoring pilots on client cameras.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeVideoMAE V2 trains a video ViT with a billion parameters; training compute is not stated in the abstract.
- DataKinetics-400 (90.0%), Kinetics-600 (89.9%) and Something-Something V1 and V2 (68.7% and 77.0%).
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 8 authors on the VideoMAE V2 paper (arXiv 2303.16727).
- Time horizonYear.
- Shelf-life signalsVideo LLMs answer questions about events directly, Video encoders become general foundation models
- Pipeline roleTier A pretrains video models; Tier C fine-tunes them on local footage; Tier D applies them to unsafe-act and process detection pilots.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
VideoPrismvisionexemplarVideoMAE V2visionexemplar
SlowFastvisionexemplar
ST-GCNvisionexemplar
X-CLIPvisionproduced byX-CLIP base patch32embeddingproduced by
VideoMAEvisionproduced by
UniFormervisionproduced by
ActionCLIPvisionproduced by
CTR-GCNvisionproduced by
Video Swin Transformervisionproduced byPoseC3Dvisionproduced byMViT (Multiscale Vision Transformers)visionproduced byViViTvisionproduced by
TCANetvisionproduced byMoViNetvisionproduced byTimeSformervisionproduced byTANet (TAM)visionproduced byX3Dvisionproduced byTPNvisionproduced by
MS-G3Dvisionproduced by
TINvisionproduced by
BMNvisionproduced byCSNvisionproduced by+18
LFB (Long-Term Feature Bank)visionproduced by
TSM (Temporal Shift Module)visionproduced byACRNvisionproduced by
BSNvisionproduced by2s-AGCNvisionproduced byS3Dvisionproduced byVideo ResNet (R3D, MC3, R(2+1)D)visionproduced byTRNvisionproduced by
Non-local Networks (video)visionproduced byLSTM mobile video object detectorvisionproduced by
I3D (Inflated 3D ConvNet)visionproduced bySSN (Structured Segment Networks)visionproduced by
TSN (Temporal Segment Networks)visionproduced byC3Dvisionproduced byTiny Video Netvisionproduced byAssembleNetvisionproduced byYouTube-8M video classifiersvisionproduced by
Intel action recognitionvisionproduced byLinks
R&D TopicVideo Action Recognition and Temporal Understanding
ModelsSlowFastMeta, visionVideoMAE V2visionVideoPrismGoogle, visionST-GCNThe Chinese University of Hong Kong, vision+38 more
SlowFastMeta, visionVideoMAE V2visionVideoPrismGoogle, visionST-GCNThe Chinese University of Hong Kong, vision+38 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 42
VideoMAENanjing University, Tencent, visionproduced byUniFormerShenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shanghai AI Lab, visionproduced byActionCLIPZhejiang University, visionproduced byC3DMeta, Dartmouth College, visionproduced byLFB (Long-Term Feature Bank)University of Texas at Austin, Meta, visionproduced byLSTM mobile video object detectorGoogle, visionproduced byMoViNetGoogle, visionproduced byMS-G3DUniversity of Sydney, SenseTime, visionproduced byMViT (Multiscale Vision Transformers)Meta, visionproduced byNon-local Networks (video)Carnegie Mellon University, Meta, visionproduced byPoseC3DThe Chinese University of Hong Kong, Shanghai AI Lab, visionproduced byS3DGoogle, UC San Diego, visionproduced by+30
SSN (Structured Segment Networks)The Chinese University of Hong Kong, ETH Zurich, visionproduced byTANet (TAM)Nanjing University, SenseTime, visionproduced byTCANetHuazhong University of Science and Technology, SenseTime, visionproduced byTiny Video NetGoogle, visionproduced byTPNThe Chinese University of Hong Kong, SenseTime, visionproduced byTSN (Temporal Segment Networks)ETH Zurich, The Chinese University of Hong Kong, visionproduced byVideo Swin TransformerMicrosoft Research, visionproduced byVideoMAE V2visionexemplarVideoPrismGoogle, visionexemplarViViTGoogle, visionproduced byX-CLIPMicrosoft AI, visionproduced byX-CLIP base patch32Microsoft AI, embeddingproduced byX3DMeta, visionproduced by2s-AGCNInstitute of Automation, Chinese Academy of Sciences, visionproduced byACRNGoogle, visionproduced byAssembleNetGoogle, visionproduced byBMNBaidu, visionproduced byBSNShanghai Jiao Tong University, visionproduced byCSNMeta, visionproduced byCTR-GCNInstitute of Automation, Chinese Academy of Sciences, visionproduced byI3D (Inflated 3D ConvNet)Google DeepMind, visionproduced byIntel action recognitionIntel, visionproduced bySlowFastMeta, visionexemplarST-GCNThe Chinese University of Hong Kong, visionexemplarTimeSformerMeta, visionproduced byTINTsinghua University, visionproduced byTRNMassachusetts Institute of Technology, visionproduced byTSM (Temporal Shift Module)Massachusetts Institute of Technology, visionproduced byYouTube-8M video classifiersGoogle, visionproduced byVideo ResNet (R3D, MC3, R(2+1)D)visionproduced byPapers
- Two-Stream Convolutional Networks for Action Recognition in VideosKaren Simonyan et al., arXiv preprint, 2014 founding
- Quo Vadis, Action Recognition? A New Model and the Kinetics DatasetJoao Carreira et al., arXiv preprint, 2017 founding
- InternVideo2: Scaling Foundation Models for Multimodal Video UnderstandingYi Wang et al., ECCV2024 (per arXiv comment), 2024 state of the art
- VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-TrainingZhan Tong et al., NeurIPS 2022 (per arXiv comment), 2022 state of the art
- Is Space-Time Attention All You Need for Video Understanding?Gedas Bertasius et al., ICML 2021 (per arXiv comment), 2021 state of the art
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2303.16727 https://arxiv.org/abs/2303.16727
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Video Action Recognition and Temporal Understanding", https://aurion.semai.id/topics/video-action-recognition/, records of October 2026.