GUI and Computer-Use Agents
Vision-language models that read screenshots and act on computer or phone interfaces by clicking and typing.
Tier Grades
Tier A: Frontier AI labs
feasible
OpenAI, Anthropic, Google and Microsoft ship computer-use models.
Tier B: Indonesian AI labs
stretch
Training an 18B screenshot model (CogAgent) with action data is a stretch; Indonesian work is not in the sources.
Tier C: Institution units such as ITB
stretch
Evaluating open computer-use models on Indonesian apps is institution-scale; training is a stretch.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Released computer-use models automate back-office screens in robotic process automation proofs of concept.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeCogAgent: 18B-parameter VLM with 1120 by 1120 input.
- DataCogAgent uses only screenshots and outperforms LLM methods that consume extracted HTML text on PC and Android navigation benchmarks (dataset sizes not in the abstract).
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 14 authors on CogAgent.
- Time horizonMonths.
- Shelf-life signalsComputer-use offerings from OpenAI, Anthropic and Google, Small open computer-use models (Microsoft Fara-7B)
- Pipeline roleTier A trains computer-use models; Tier C tests them on Indonesian apps and government portals; Tier D automates client workflows.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
Fara-7Bmultimodal LLMexemplar
computer-use-previewmultimodal LLMexemplar
CogAgent chatmultimodal LLMexemplarGLM-5V-Turbomultimodal LLMproduced byGLM-4.5Vmultimodal LLMproduced byMagma-8Bmultimodal LLMproduced by
Claude 3.5 Sonnet (Oct '24)multimodal LLMproduced by
gemini-2.5-computer-use-preview-10-2025LLMproduced byLinks
R&D TopicGUI and Computer-Use Agents
ModelsFara-7BMicrosoft AI, multimodal LLMcomputer-use-previewOpenAI, multimodal LLMCogAgent chatZ.ai (Zhipu), multimodal LLMClaude 3.5 Sonnet (Oct '24)Anthropic, multimodal LLM+4 more
Fara-7BMicrosoft AI, multimodal LLMcomputer-use-previewOpenAI, multimodal LLMCogAgent chatZ.ai (Zhipu), multimodal LLMClaude 3.5 Sonnet (Oct '24)Anthropic, multimodal LLM+4 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 8
GLM-4.5VZ.ai (Zhipu), multimodal LLMproduced byClaude 3.5 Sonnet (Oct '24)Anthropic, multimodal LLMproduced byCogAgent chatZ.ai (Zhipu), Tsinghua University, multimodal LLMexemplargemini-2.5-computer-use-preview-10-2025Google DeepMind, LLMproduced byGLM-5V-TurboZ.ai (Zhipu), multimodal LLMproduced byFara-7BMicrosoft AI, multimodal LLMexemplarMagma-8BMicrosoft AI, multimodal LLMproduced bycomputer-use-previewOpenAI, multimodal LLMexemplarPapers
- WebGPT: Browser-assisted question-answering with human feedbackReiichiro Nakano et al., arXiv preprint, 2021 founding
- UI-TARS: Pioneering Automated GUI Interaction with Native AgentsYujia Qin et al., arXiv preprint, 2025 state of the art
- GPT-4V(ision) is a Generalist Web Agent, if GroundedBoyuan Zheng et al., arXiv preprint, 2024 state of the art
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer EnvironmentsTianbao Xie et al., arXiv preprint, 2024 state of the art
- GUI Agents: A SurveyDang Nguyen et al., ACL 2025 (per arXiv comment), 2024 survey
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2312.08914 https://arxiv.org/abs/2312.08914
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "GUI and Computer-Use Agents", https://aurion.semai.id/topics/gui-and-computer-use-agents/, records of October 2026.