Document, OCR and Chart Understanding
Vision-language models that read text-rich images such as scanned documents, forms, charts and screenshots without a separate OCR pipeline.
Tier Grades
Tier A: Frontier AI labs
feasible
Google, Microsoft, Mistral and OpenBMB release document-capable VLMs.
Tier B: Indonesian AI labs
feasible
Fine-tuning an open document VLM on Indonesian forms is within reach; Indonesian lab capacity is not in the sources.
Tier C: Institution units such as ITB
feasible
Building Indonesian document test sets is institution-scale; pretraining an 18B model (CogAgent) is not needed.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Reading invoices, forms and charts with an open phone-sized VLM (MiniCPM-V) is a common proof of concept.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeCogAgent: 18B parameters with 1120 by 1120 input to read tiny text; MiniCPM-V targets end-side devices such as phones.
- DataCogAgent reports state of the art on text-rich benchmarks including TextVQA, ChartQA, InfoVQA and DocVQA; dataset sizes are not in the abstracts.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 23 authors on MiniCPM-V; 14 on CogAgent.
- Time horizonMonths.
- Shelf-life signalsSmall on-device document VLMs (MiniCPM-V on phones), OCR-free reading replacing OCR plus layout pipelines in document RAG
- Pipeline roleTier A and open labs release document VLMs; Tier C builds Indonesian document benchmarks; Tier D applies them to invoices, forms and reports.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Google
OpenBMB (Tsinghua NLP Lab and ModelBest)
Microsoft AI
Z.ai (Zhipu)
Mistral AI
Hugging Face
Adept AI
Shanghai AI LabLinked models
NVIDIA Nemotron Nano 12B v2 VLmultimodal LLMexemplarPixtral Largemultimodal LLMexemplarMiniCPM-Vmultimodal LLMexemplarFlorence-2multimodal LLMexemplarGLM-4.6Vmultimodal LLMproduced by
Qwen3-VL-235B-A22B-Instructmultimodal LLMproduced byMiniCPM-V 4.5multimodal LLMproduced byGLM-4.5Vmultimodal LLMproduced byPaliGemma 2 28B pt 896multimodal LLMproduced byPaliGemma 2multimodal LLMproduced byPaliGemma 2 10B pt 448multimodal LLMproduced byPaliGemma 2 3B pt 224multimodal LLMproduced byPhi-3.5-vision-instructmultimodal LLMproduced byMiniCPM-V 2.6multimodal LLMproduced byInternLM-XComposer 2.5 7Bmultimodal LLMproduced byFlorence-2 basemultimodal LLMproduced byFlorence-2 largemultimodal LLMproduced byMiniCPM-Llama3-V 2.5multimodal LLMproduced byPaliGemma 3B mix 448multimodal LLMproduced byPaliGemmamultimodal LLMproduced byPaliGemma 3B pt 224multimodal LLMproduced byIdeficsmultimodal LLMproduced byFuyu-8Bmultimodal LLMproduced byKosmos-2multimodal LLMproduced byLinks
R&D TopicDocument, OCR and Chart Understanding
ModelsPixtral LargeMistral AI, multimodal LLMFlorence-2Microsoft AI, multimodal LLMNVIDIA Nemotron Nano 12B v2 VLNVIDIA, multimodal LLMMiniCPM-VOpenBMB (Tsinghua NLP Lab and ModelBest), multimodal LLM+22 more
Pixtral LargeMistral AI, multimodal LLMFlorence-2Microsoft AI, multimodal LLMNVIDIA Nemotron Nano 12B v2 VLNVIDIA, multimodal LLMMiniCPM-VOpenBMB (Tsinghua NLP Lab and ModelBest), multimodal LLM+22 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 26
GLM-4.5VZ.ai (Zhipu), multimodal LLMproduced byFlorence-2Microsoft AI, multimodal LLMexemplarMiniCPM-VOpenBMB (Tsinghua NLP Lab and ModelBest), multimodal LLMexemplarPaliGemma 2Google, multimodal LLMproduced byFlorence-2 baseMicrosoft AI, multimodal LLMproduced byFlorence-2 largeMicrosoft AI, multimodal LLMproduced byIdeficsHugging Face, multimodal LLMproduced byPaliGemmaGoogle, multimodal LLMproduced byPhi-3.5-vision-instructMicrosoft AI, multimodal LLMproduced byGLM-4.6VZ.ai (Zhipu), multimodal LLMproduced byKosmos-2Microsoft AI, multimodal LLMproduced byMiniCPM-Llama3-V 2.5OpenBMB (Tsinghua NLP Lab and ModelBest), multimodal LLMproduced by+14
MiniCPM-V 2.6OpenBMB (Tsinghua NLP Lab and ModelBest), multimodal LLMproduced byMiniCPM-V 4.5OpenBMB (Tsinghua NLP Lab and ModelBest), multimodal LLMproduced byMiniCPM-V 4.6 1.3BOpenBMB (Tsinghua NLP Lab and ModelBest), multimodal LLMproduced byPaliGemma 2 10B pt 448Google, multimodal LLMproduced byPaliGemma 2 28B pt 896Google, multimodal LLMproduced byPaliGemma 2 3B pt 224Google, multimodal LLMproduced byPaliGemma 3B mix 448Google, multimodal LLMproduced byPaliGemma 3B pt 224Google, multimodal LLMproduced byPixtral LargeMistral AI, multimodal LLMexemplarQwen3-VL-235B-A22B-InstructAlibaba (Qwen), multimodal LLMproduced byFuyu-8BAdept AI, multimodal LLMproduced byGemini 3 FlashGoogle, multimodal LLMused byInternLM-XComposer 2.5 7BShanghai AI Lab, multimodal LLMproduced byNVIDIA Nemotron Nano 12B v2 VLNVIDIA, multimodal LLMexemplarPapers
- DocVQA: A Dataset for VQA on Document ImagesMinesh Mathew et al., WACV 2021 (per arXiv comment), 2020 founding
- Qwen2.5-VL Technical ReportShuai Bai et al., arXiv preprint, 2025 state of the art
- DeepSeek-OCR: Contexts Optical CompressionHaoran Wei et al., arXiv preprint, 2025 state of the art
- mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document UnderstandingAnwen Hu et al., arXiv preprint, 2024 state of the art
- OCRBench: On the Hidden Mystery of OCR in Large Multimodal ModelsYuliang Liu et al., arXiv preprint, 2023 state of the art
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2408.01800 https://arxiv.org/abs/2408.01800
- arxiv.org/abs/2312.08914 https://arxiv.org/abs/2312.08914
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Document, OCR and Chart Understanding", https://aurion.semai.id/topics/document-ocr-and-chart-understanding/, records of October 2026.