Omni-Modal and Unified Multimodal Models
Single models that take text, images, audio and video and produce text and speech, or that both understand and generate images in one transformer.
Tier Grades
Tier A: Frontier AI labs
feasible
Alibaba, DeepSeek, ByteDance and Xiaomi train omni and unified models.
Tier B: Indonesian AI labs
stretch
Training across text, image, audio and video at once is a data and compute stretch; Indonesian work is not in the sources.
Tier C: Institution units such as ITB
stretch
Studying released 7B omni models is institution-scale; training one is a stretch.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Released 7B omni models (Qwen2.5-Omni) support voice and vision assistant proofs of concept.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeQwen2.5-Omni: Thinker-Talker design that streams text and speech; Janus: one autoregressive transformer with decoupled visual encoders; compute is not in the abstracts.
- DataQwen2.5-Omni interleaves audio and video with time-aligned position embeddings (TMRoPE); dataset sizes are not in the abstracts.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 14 authors on Qwen2.5-Omni; 11 on Janus.
- Time horizonMonths.
- Shelf-life signalsOmni releases from Alibaba, Xiaomi, OpenBMB and NVIDIA within the same year, Unified understanding and generation (Janus, BAGEL)
- Pipeline roleTier A trains omni models; Tier C evaluates them in Indonesian; Tier D builds multimodal assistants on released checkpoints.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
BAGEL (BAGEL-7B-MoT)multimodal LLMexemplar
Qwen2.5-Omni-7Bmultimodal LLMexemplar
MiniCPM-omultimodal LLMexemplar
Janusmultimodal LLMexemplar
MiMo-V2.6-Flashmultimodal LLMproduced byMiMo-V2.5multimodal LLMproduced by
HyperCLOVA X SEED 8B Omnimultimodal LLMproduced by
Nemotron 3 Nano Omnimultimodal LLMproduced byMiMo-V2-Omnimultimodal LLMproduced byMiniCPM-o 4.5multimodal LLMproduced byQwen3-Omni-30B-A3B-Instructmultimodal LLMproduced byJanus-Pro 1Bmultimodal LLMproduced byJanus-Pro 7Bmultimodal LLMproduced byMiniCPM-o 2.6multimodal LLMproduced byJanusFlow 1.3Bmultimodal LLMproduced byJanus 1.3Bmultimodal LLMproduced by
ImageBindembeddingproduced byqwen3.8-omni-flashmultimodal LLMproduced byqwen3.8-omni-flash-realtimemultimodal LLMproduced by
Nova 2.0 OmniLLMproduced byLinks
R&D TopicOmni-Modal and Unified Multimodal Models
ModelsBAGEL (BAGEL-7B-MoT)ByteDance, multimodal LLMQwen2.5-Omni-7BAlibaba (Qwen), multimodal LLMJanusDeepSeek, multimodal LLMMiniCPM-oOpenBMB (Tsinghua NLP Lab and ModelBest), multimodal LLM+16 more
BAGEL (BAGEL-7B-MoT)ByteDance, multimodal LLMQwen2.5-Omni-7BAlibaba (Qwen), multimodal LLMJanusDeepSeek, multimodal LLMMiniCPM-oOpenBMB (Tsinghua NLP Lab and ModelBest), multimodal LLM+16 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 20
MiMo-V2.6-FlashXiaomi, multimodal LLMproduced byJanusDeepSeek, multimodal LLMexemplarMiMo-V2.5Xiaomi, multimodal LLMproduced byMiMo-V2-OmniXiaomi, multimodal LLMproduced byMiniCPM-oOpenBMB (Tsinghua NLP Lab and ModelBest), multimodal LLMexemplarNova 2.0 OmniAmazon, LLMproduced byqwen3.8-omni-flashAlibaba (Qwen), multimodal LLMproduced byqwen3.8-omni-flash-realtimeAlibaba (Qwen), multimodal LLMproduced byBAGEL (BAGEL-7B-MoT)ByteDance, multimodal LLMexemplarImageBindMeta, embeddingproduced byJanus 1.3BDeepSeek, multimodal LLMproduced byJanus-Pro 1BDeepSeek, multimodal LLMproduced by+8
Janus-Pro 7BDeepSeek, multimodal LLMproduced byJanusFlow 1.3BDeepSeek, multimodal LLMproduced byMiniCPM-o 2.6OpenBMB (Tsinghua NLP Lab and ModelBest), multimodal LLMproduced byMiniCPM-o 4.5OpenBMB (Tsinghua NLP Lab and ModelBest), multimodal LLMproduced byNemotron 3 Nano OmniNVIDIA, multimodal LLMproduced byHyperCLOVA X SEED 8B OmniNaver, multimodal LLMproduced byQwen2.5-Omni-7BAlibaba (Qwen), multimodal LLMexemplarQwen3-Omni-30B-A3B-InstructAlibaba (Qwen), multimodal LLMproduced byPapers
- NExT-GPT: Any-to-Any Multimodal LLMShengqiong Wu et al., ICML 2024 (per arXiv comment), 2023 founding
- Qwen2.5-Omni Technical ReportJin Xu et al., arXiv preprint, 2025 state of the art
- GPT-4o System CardOpenAI et al., arXiv preprint, 2024 state of the art
- Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and ActionJiasen Lu et al., arXiv preprint, 2023 state of the art
- MM-LLMs: Recent Advances in MultiModal Large Language ModelsDuzhen Zhang et al., ACL2024 (per arXiv comment), 2024 survey
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2503.20215 https://arxiv.org/abs/2503.20215
- arxiv.org/abs/2410.13848 https://arxiv.org/abs/2410.13848
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Omni-Modal and Unified Multimodal Models", https://aurion.semai.id/topics/omni-modal-and-unified-multimodal-models/, records of October 2026.