Visual Instruction Tuning
Connecting a vision encoder to a language model and training it on image-grounded instructions to make an open vision-language assistant.
Tier Grades
Tier A: Frontier AI labs
feasible
Google, Alibaba, Shanghai AI Lab and Ai2 train open VLMs.
Tier B: Indonesian AI labs
feasible
LLaVA's recipe of an open encoder, an open LLM and generated instruction data needs modest compute; Indonesian VLM work is not in the sources.
Tier C: Institution units such as ITB
feasible
LLaVA came from a 4-author team.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Open VLMs answer questions about client images in inspection and document proofs of concept.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeLLaVA connects a vision encoder and an LLM trained end to end; compute is not given in the abstract.
- DataMultimodal instruction data generated with language-only GPT-4; LLaVA scores 85.1% relative to GPT-4 on a synthetic multimodal instruction set and 92.53% on Science QA with GPT-4.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 4 authors on LLaVA.
- Time horizonMonths.
- Shelf-life signalsOpen VLM families updated several times a year (LLaVA, InternVL, Molmo, Qwen-VL), Small VLMs under 2B for phones (SmolVLM, FastVLM)
- Pipeline roleTier A and open labs release VLMs; Tier B and Tier C tune them on Indonesian images and text; Tier D applies them to inspection and visual question answering.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
Molmomultimodal LLMexemplar
Ideficsmultimodal LLMexemplar
InternVLmultimodal LLMexemplar
LLaVAmultimodal LLMexemplarIntern-S1-Promultimodal LLMproduced by
Ling 3.0 Flash VL (Ling-3.0-flash-VL)multimodal LLMproduced by
DeepSeek-V4-Flash Vision-Expmultimodal LLMproduced byIntern-S2-397Bmultimodal LLMproduced by
LFM2.5-VL-3Bmultimodal LLMproduced by
Step-3.7-Flashmultimodal LLMproduced by
Gemini Robotics-ER 1.6multimodal LLMproduced by
Phi-4-reasoning-vision-15Bmultimodal LLMproduced byStep3-VL-10Bmultimodal LLMproduced by
Cosmos Reason 2multimodal LLMproduced byMolmo 2multimodal LLMproduced byMolmo2-8Bmultimodal LLMproduced by
GLM-4.6Vmultimodal LLMproduced by
Qwen-SEA-LION-v4-8B-VLmultimodal LLMproduced by
Ministral 3 14B Instruct (25.12)multimodal LLMproduced byNVIDIA Nemotron Nano 12B v2 VLmultimodal LLMproduced by
Qwen3-VL-8B-Instructmultimodal LLMproduced byQwen3-VL-235B-A22B-Instructmultimodal LLMproduced by
Keye-VL 1.5 8Bmultimodal LLMproduced by
FastVLM 0.5Bmultimodal LLMproduced by+81
FastVLM 7Bmultimodal LLMproduced byInternVL3.5 241B A28Bmultimodal LLMproduced byInternVL3.5 2Bmultimodal LLMproduced byInternVL3.5 8Bmultimodal LLMproduced by
MiniCPM-V 4.5multimodal LLMproduced byGLM-4.5Vmultimodal LLMproduced byKeye-VLmultimodal LLMproduced byKeye-VL 8B Previewmultimodal LLMproduced byGemma 3n E2Bedge LLMproduced byGemma 3n E4Bedge LLMproduced by
Kimi-VLmultimodal LLMproduced byKimi-VL-A3B-Instructmultimodal LLMproduced byKimi-VL-A3B-Thinkingmultimodal LLMproduced byGemini Robotics-ERmultimodal LLMproduced by
Aya Visionmultimodal LLMproduced byAya Vision 32Bmultimodal LLMproduced byAya Vision 8Bmultimodal LLMproduced bySmolVLM2 256M Video Instructmultimodal LLMproduced bySmolVLM2 500M Video Instructmultimodal LLMproduced bySmolVLM2 2.2B Instructmultimodal LLMproduced byInternVideo2.5 Chat 8Bmultimodal LLMproduced byQVQ-72B-Previewreasoningproduced byFastVLMmultimodal LLMproduced by
PaliGemma 2 28B pt 896multimodal LLMproduced byPaliGemma 2multimodal LLMproduced byPaliGemma 2 10B pt 448multimodal LLMproduced byPaliGemma 2 3B pt 224multimodal LLMproduced bySmolVLMmultimodal LLMproduced bySmolVLM Instructmultimodal LLMproduced byMolmo 7B-Dmultimodal LLMproduced byQwen2-VL 2Bmultimodal LLMproduced byPhi-3.5-vision-instructmultimodal LLMproduced by
LLaVA-OneVision Qwen2 7Bmultimodal LLMproduced by
mPLUG-Owl3 7Bmultimodal LLMproduced by
xGen-MM (BLIP-3) phi3 mini instructmultimodal LLMproduced byIdefics3 8B Llama3multimodal LLMproduced byMiniCPM-V 2.6multimodal LLMproduced by
Prismmultimodal LLMproduced byInternLM-XComposer 2.5 7Bmultimodal LLMproduced byInternVL2 8Bmultimodal LLMproduced byFlorence-2 basemultimodal LLMproduced byFlorence-2 largemultimodal LLMproduced byLLaVA-NeXT-Video 7Bmultimodal LLMproduced byGLM-4V-9Bmultimodal LLMproduced byMiniCPM-Llama3-V 2.5multimodal LLMproduced byLLaVA-Med v1.5 (Mistral 7B)multimodal LLMproduced byPaliGemma 3B mix 448multimodal LLMproduced byPaliGemmamultimodal LLMproduced byPaliGemma 3B pt 224multimodal LLMproduced bySeaLMMM 7B v0.1multimodal LLMproduced byIdefics2 8Bmultimodal LLMproduced byLLaVA-NeXT (1.6) Mistral 7Bmultimodal LLMproduced byMiniCPM-Vmultimodal LLMproduced by
Yi-VLmultimodal LLMproduced byYi-VL 34Bmultimodal LLMproduced byYi-VL 6Bmultimodal LLMproduced byCogAgent chatmultimodal LLMproduced byLLaVA-1.5 7Bmultimodal LLMproduced byCogVLM chatmultimodal LLMproduced byFlorence-2multimodal LLMproduced by
Fuyu-8Bmultimodal LLMproduced byCogVLMmultimodal LLMproduced byKosmos-2multimodal LLMproduced byQwen-VL-Chatmultimodal LLMproduced bymPLUG-Owlmultimodal LLMproduced by
MiniGPT-4multimodal LLMproduced byBLIP-2 Flan-T5 XLmultimodal LLMproduced byBLIP-2 OPT 2.7Bmultimodal LLMproduced byBLIP-2multimodal LLMproduced byBLIP image captioning largemultimodal LLMproduced byBLIP VQA basemultimodal LLMproduced byOFAmultimodal LLMproduced byBLIPmultimodal LLMproduced by
ViLT (VQA fine-tune)multimodal LLMproduced byGemma 3nedge LLMproduced byMoondream 1.8Bedge LLMproduced by
LLM-jp-4-VL-9Bmultimodal LLMproduced byEagle 2.5multimodal LLMproduced byMiniCPM-V 4.6 1.3Bmultimodal LLMproduced byQwen2.5-VLmultimodal LLMproduced by
UnifoLM-VLM-Basemultimodal LLMproduced byLinks
R&D TopicVisual Instruction Tuning
ModelsMolmoAllen AI, multimodal LLMIdeficsHugging Face, multimodal LLMLLaVAUniversity of Wisconsin-Madison, multimodal LLMInternVLShanghai AI Lab, multimodal LLM+101 more
MolmoAllen AI, multimodal LLMIdeficsHugging Face, multimodal LLMLLaVAUniversity of Wisconsin-Madison, multimodal LLMInternVLShanghai AI Lab, multimodal LLM+101 moreWho Runs Itnone recorded
Models 105
Gemini Robotics-ER 1.6Google DeepMind, Boston Dynamics, multimodal LLMproduced byGemma 3nGoogle DeepMind, edge LLMproduced byGLM-4.5VZ.ai (Zhipu), multimodal LLMproduced byLLaVAUniversity of Wisconsin-Madison, Microsoft AI, multimodal LLMexemplarQwen-SEA-LION-v4-8B-VLAI Singapore, multimodal LLMproduced bySmolVLMHugging Face, multimodal LLMproduced byCogAgent chatZ.ai (Zhipu), Tsinghua University, multimodal LLMproduced byCogVLMZ.ai (Zhipu), Tsinghua University, multimodal LLMproduced byCosmos Reason 2NVIDIA, multimodal LLMproduced byDeepSeek-V4-Flash Vision-ExpDeepSeek, multimodal LLMproduced byFlorence-2Microsoft AI, multimodal LLMproduced byGemma 3n E2BGoogle DeepMind, edge LLMproduced by+93
Gemma 3n E4BGoogle DeepMind, edge LLMproduced byInternVLShanghai AI Lab, multimodal LLMexemplarKimi-VL-A3B-ThinkingMoonshot AI, multimodal LLMproduced byMiniCPM-VOpenBMB (Tsinghua NLP Lab and ModelBest), multimodal LLMproduced byPaliGemma 2Google, multimodal LLMproduced byAya VisionCohere Labs, multimodal LLMproduced byCogVLM chatZ.ai (Zhipu), Tsinghua University, multimodal LLMproduced byFastVLMApple, multimodal LLMproduced byFlorence-2 baseMicrosoft AI, multimodal LLMproduced byFlorence-2 largeMicrosoft AI, multimodal LLMproduced byGemini Robotics-ERGoogle DeepMind, multimodal LLMproduced byIdeficsHugging Face, multimodal LLMexemplarKeye-VLKuaishou, multimodal LLMproduced byKimi-VLMoonshot AI, multimodal LLMproduced byKimi-VL-A3B-InstructMoonshot AI, multimodal LLMproduced byLLaVA-NeXT-Video 7BUniversity of Wisconsin-Madison, Microsoft AI, multimodal LLMproduced byMolmo2-8BAllen AI, multimodal LLMproduced byPaliGemmaGoogle, multimodal LLMproduced byPhi-3.5-vision-instructMicrosoft AI, multimodal LLMproduced byPrismPaige, multimodal LLMproduced byQwen2-VL 2BAlibaba (Qwen), multimodal LLMproduced bySmolVLM InstructHugging Face, multimodal LLMproduced bySmolVLM2 2.2B InstructHugging Face, multimodal LLMproduced bySmolVLM2 256M Video InstructHugging Face, multimodal LLMproduced bySmolVLM2 500M Video InstructHugging Face, multimodal LLMproduced byStep-3.7-FlashStepFun, multimodal LLMproduced byUnifoLM-VLM-BaseUnitree Robotics, multimodal LLMproduced byAya Vision 32BCohere Labs, multimodal LLMproduced byAya Vision 8BCohere Labs, multimodal LLMproduced byBLIPSalesforce, multimodal LLMproduced byBLIP-2Salesforce, multimodal LLMproduced byEagle 2.5NVIDIA, multimodal LLMproduced byFastVLM 0.5BApple, multimodal LLMproduced byFastVLM 7BApple, multimodal LLMproduced byGLM-4.6VZ.ai (Zhipu), multimodal LLMproduced byIntern-S2-397BShanghai AI Lab, multimodal LLMproduced byKeye-VL 1.5 8BKuaishou, multimodal LLMproduced byKeye-VL 8B PreviewKuaishou, multimodal LLMproduced byKosmos-2Microsoft AI, multimodal LLMproduced byLing 3.0 Flash VL (Ling-3.0-flash-VL)inclusionAI, multimodal LLMproduced byLLaVA-1.5 7BUniversity of Wisconsin-Madison, Microsoft AI, multimodal LLMproduced byLLaVA-Med v1.5 (Mistral 7B)Microsoft AI, multimodal LLMproduced byLLaVA-NeXT (1.6) Mistral 7BUniversity of Wisconsin-Madison, Microsoft AI, multimodal LLMproduced byLLM-jp-4-VL-9BNII LLM-jp, multimodal LLMproduced byMiniCPM-Llama3-V 2.5OpenBMB (Tsinghua NLP Lab and ModelBest), multimodal LLMproduced byMiniCPM-V 2.6OpenBMB (Tsinghua NLP Lab and ModelBest), multimodal LLMproduced byMiniCPM-V 4.5OpenBMB (Tsinghua NLP Lab and ModelBest), multimodal LLMproduced byMiniCPM-V 4.6 1.3BOpenBMB (Tsinghua NLP Lab and ModelBest), multimodal LLMproduced byMolmo 2Allen AI, multimodal LLMproduced byMolmo 7B-DAllen AI, multimodal LLMproduced byPaliGemma 2 10B pt 448Google, multimodal LLMproduced byPaliGemma 2 28B pt 896Google, multimodal LLMproduced byPaliGemma 2 3B pt 224Google, multimodal LLMproduced byPaliGemma 3B mix 448Google, multimodal LLMproduced byPaliGemma 3B pt 224Google, multimodal LLMproduced byQwen2.5-VLmultimodal LLMproduced byQwen3-VL-235B-A22B-InstructAlibaba (Qwen), multimodal LLMproduced bySeaLMMM 7B v0.1Alibaba DAMO, multimodal LLMproduced byYi-VL01.AI, multimodal LLMproduced byBLIP image captioning largeSalesforce, multimodal LLMproduced byBLIP VQA baseSalesforce, multimodal LLMproduced byBLIP-2 Flan-T5 XLSalesforce, multimodal LLMproduced byBLIP-2 OPT 2.7BSalesforce, multimodal LLMproduced byFuyu-8BAdept AI, multimodal LLMproduced byIdefics2 8BHugging Face, multimodal LLMproduced byIdefics3 8B Llama3Hugging Face, multimodal LLMproduced byInternLM-XComposer 2.5 7BShanghai AI Lab, multimodal LLMproduced byInternVideo2.5 Chat 8BShanghai AI Lab, multimodal LLMproduced byInternVL2 8BShanghai AI Lab, multimodal LLMproduced byInternVL3.5 241B A28BShanghai AI Lab, multimodal LLMproduced byInternVL3.5 2BShanghai AI Lab, multimodal LLMproduced byInternVL3.5 8BShanghai AI Lab, multimodal LLMproduced byLFM2.5-VL-3BLiquid AI, multimodal LLMproduced byLLaVA-OneVision Qwen2 7BByteDance, multimodal LLMproduced byMinistral 3 14B Instruct (25.12)Mistral AI, multimodal LLMproduced byMolmoAllen AI, multimodal LLMexemplarMoondream 1.8Bedge LLMproduced bymPLUG-OwlAlibaba DAMO, multimodal LLMproduced bymPLUG-Owl3 7BAlibaba DAMO, multimodal LLMproduced byNVIDIA Nemotron Nano 12B v2 VLNVIDIA, multimodal LLMproduced byPhi-4-reasoning-vision-15BMicrosoft AI, multimodal LLMproduced byQVQ-72B-PreviewAlibaba (Qwen), reasoningproduced byQwen3-VL-8B-InstructAlibaba (Qwen), multimodal LLMproduced byYi-VL 34B01.AI, multimodal LLMproduced byYi-VL 6B01.AI, multimodal LLMproduced byGLM-4V-9BZ.ai (Zhipu), multimodal LLMproduced byIntern-S1-ProShanghai AI Lab, multimodal LLMproduced byMiniGPT-4King Abdullah University of Science and Technology, multimodal LLMproduced byOFAAlibaba DAMO, multimodal LLMproduced byQwen-VL-ChatAlibaba (Qwen), multimodal LLMproduced byStep3-VL-10BStepFun, multimodal LLMproduced byViLT (VQA fine-tune)Naver, multimodal LLMproduced byxGen-MM (BLIP-3) phi3 mini instructSalesforce, multimodal LLMproduced byPapers
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac et al., arXiv preprint, 2022 founding
- Visual Instruction TuningHaotian Liu et al., NeurIPS 2023 (per arXiv comment), 2023 founding
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any ResolutionPeng Wang et al., arXiv preprint, 2024 state of the art
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMsShengbang Tong et al., NeurIPS 2024 (per arXiv comment), 2024 state of the art
- A Survey on Multimodal Large Language ModelsShukang Yin et al., arXiv preprint, 2023 survey
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2304.08485 https://arxiv.org/abs/2304.08485
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Visual Instruction Tuning", https://aurion.semai.id/topics/visual-instruction-tuning/, records of October 2026.