Medical Vision-Language Models
Multimodal models that read medical images and text to answer clinical questions and draft grounded radiology or pathology reports.
Tier Grades
Tier A: Frontier AI labs
feasible
Google and Microsoft build medical VLMs with large clinical teams (MedGemma: 81 authors).
Tier B: Indonesian AI labs
stretch
Clinical image data and validation are a stretch for a lab; Indonesian medical VLM work is not in the sources.
Tier C: Institution units such as ITB
feasible
Hospital-linked institution units can fine-tune a released 4B MedGemma on local images.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
stretch
Proofs of concept are possible on MedGemma 4B, but clinical validation of the kind MAIRA-2 reports makes value proofs a stretch.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeMedGemma: medical vision-language models built on Gemma 3 4B and 27B; compute is not in the abstract.
- DataMedGemma: 2.6 to 10% gain on medical multimodal question answering and 15.5 to 18.1% on chest X-ray finding classification; MAIRA-2 adds finding localisation and the RadFact evaluation.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 81 authors on MedGemma; 21 on MAIRA-2.
- Time horizonOne to two years.
- Shelf-life signalsOpen medical models on general bases (MedGemma 4B) replacing closed ones, Grounded report generation with per-sentence checking (MAIRA-2, RadFact)
- Pipeline roleTier A releases medical VLMs; Tier C validates them with Indonesian hospitals; Tier D builds supervised drafting aids with clinical partners.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
MAIRA-2multimodal LLMexemplarLLaVA-Med v1.5 (Mistral 7B)multimodal LLMexemplar
MedGemma 4Bmultimodal LLMexemplarMedGemma 1.5 4Bmultimodal LLMproduced byGemma 3 4Bmultimodal LLMused by
Prismmultimodal LLMproduced byBiomedCLIPembeddingused byAMIEmultimodal LLMproduced byMedGemmamultimodal LLMproduced byMedGemma 27B multimodalmultimodal LLMproduced by
CheXagentmultimodal LLMproduced by
Vicuna-7B-v1.5LLMused byLinks
R&D TopicMedical Vision-Language Models
ModelsMedGemma 4BGoogle, multimodal LLMMAIRA-2Microsoft AI, multimodal LLMLLaVA-Med v1.5 (Mistral 7B)Microsoft AI, multimodal LLMAMIEGoogle, multimodal LLM+8 more
MedGemma 4BGoogle, multimodal LLMMAIRA-2Microsoft AI, multimodal LLMLLaVA-Med v1.5 (Mistral 7B)Microsoft AI, multimodal LLMAMIEGoogle, multimodal LLM+8 more
Models 12
MedGemma 4BGoogle, multimodal LLMexemplarAMIEGoogle, Google DeepMind, multimodal LLMproduced byMedGemmaGoogle, multimodal LLMproduced byMedGemma 1.5 4BGoogle, multimodal LLMproduced byGemma 3 4BGoogle, multimodal LLMused byMAIRA-2Microsoft AI, multimodal LLMexemplarMedGemma 27B multimodalGoogle, multimodal LLMproduced byPrismPaige, multimodal LLMproduced byCheXagentStanford University, multimodal LLMproduced byLLaVA-Med v1.5 (Mistral 7B)Microsoft AI, multimodal LLMexemplarVicuna-7B-v1.5LMSYS, LLMused byBiomedCLIPMicrosoft AI, embeddingused byPapers
- Contrastive Learning of Medical Visual Representations from Paired Images and TextYuhao Zhang et al., arXiv preprint, 2020 founding
- BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairsSheng Zhang et al., arXiv preprint, 2023 state of the art
- LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One DayChunyuan Li et al., arXiv preprint, 2023 state of the art
- Towards Generalist Biomedical AITao Tu et al., arXiv preprint, 2023 state of the art
- Med-Flamingo: a Multimodal Medical Few-shot LearnerMichael Moor et al., arXiv preprint, 2023 state of the art
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2507.05201 https://arxiv.org/abs/2507.05201
- arxiv.org/abs/2406.04449 https://arxiv.org/abs/2406.04449
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Medical Vision-Language Models", https://aurion.semai.id/topics/medical-vision-language-models/, records of October 2026.