Open-Vocabulary Object Detection and Grounding
Detectors and segmenters that find objects named in free text, including classes never labelled in training.
Tier Grades
Tier A: Frontier AI labs
feasible
Grounded pretraining on large image-text data, as in Grounding DINO, is frontier and big-tech work.
Tier B: Indonesian AI labs
stretch
Pretraining a grounding model is a stretch for an Indonesian lab; adapting a released one with local labels is feasible.
Tier C: Institution units such as ITB
feasible
Evaluating zero-shot transfer on COCO and ODinW, as the 12-author Grounding DINO paper does, fits a university unit.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Zero-shot detection at 52.5 COCO AP lets a delivery team show a detector on client classes without labelling first.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeTraining compute is not in the sources read.
- DataGrounding DINO reaches 52.5 AP zero-shot on COCO without COCO training data and 26.1 mean AP on ODinW.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 12 authors on the Grounding DINO paper (arXiv 2303.05499).
- Time horizonMonths to a year.
- Shelf-life signalsMultimodal LLMs that output boxes directly match dedicated open-set detectors, Real-time open-vocabulary detectors reach closed-set accuracy
- Pipeline roleTier A pretrains grounded detectors; Tier C tests zero-shot transfer on local classes; Tier D uses them to label client images and to demo detection without training data; Tier B adds Indonesian object vocabularies.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
YOLOEvisionexemplar
YOLO-Worldvisionexemplar
OWLv2 L/14visionexemplar
Grounding DINOvisionexemplar
LocateAnything 3Bvisionproduced by
Falcon Perceptionvisionproduced byWeDetectvisionproduced by
SAM 3visionproduced by
LLMDetvisionproduced by
Florence-2 basemultimodal LLMproduced byFlorence-2 largemultimodal LLMproduced by
OmDet-Turbovisionproduced by
MM-Grounding-DINOvisionproduced byFlorence-2multimodal LLMproduced byKosmos-2multimodal LLMproduced byOWLv2 B/16 ensemblevisionproduced by
Recognize Anything (RAM)visionproduced by
CAT-Segvisionproduced by
SAN (Side Adapter Network)visionproduced by
X-Decodervisionproduced byF-VLMvisionproduced byOWL-ViTvisionproduced byOWL-ViT B/32visionproduced byDeticvisionproduced byLinks
R&D TopicOpen-Vocabulary Object Detection and Grounding
ModelsYOLOETsinghua University, visionYOLO-WorldTencent, visionGrounding DINOIDEA Research (International Digital Economy Academy), visionOWLv2 L/14Google, vision+24 more
YOLOETsinghua University, visionYOLO-WorldTencent, visionGrounding DINOIDEA Research (International Digital Economy Academy), visionOWLv2 L/14Google, vision+24 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 28
Florence-2Microsoft AI, multimodal LLMproduced bySAM 3Meta, visionproduced byFlorence-2 baseMicrosoft AI, multimodal LLMproduced byFlorence-2 largeMicrosoft AI, multimodal LLMproduced byOWL-ViTGoogle, visionproduced byX-DecoderUniversity of Wisconsin-Madison, UCLA, Microsoft Research, visionproduced byKosmos-2Microsoft AI, multimodal LLMproduced byOWL-ViT B/32Google, visionproduced byOWLv2 B/16 ensembleGoogle, visionproduced byCAT-SegKorea University, Google, visionproduced by
GLIPMicrosoft Research, UCLA, visionproduced byLLMDetSun Yat-sen University, Alibaba Group, visionproduced by+16
OWLv2 L/14Google, visionexemplarRecognize Anything (RAM)OPPO, IDEA Research (International Digital Economy Academy), visionproduced bySAN (Side Adapter Network)Huazhong University of Science and Technology, Microsoft Research, visionproduced by
CLIPSegUniversity of Gottingen, visionproduced byDeepMAC Mask R-CNNGoogle, visionproduced byDeticMeta, visionproduced byF-VLMGoogle, visionproduced byFalcon PerceptionTechnology Innovation Institute (TII), visionproduced byGrounding DINOIDEA Research (International Digital Economy Academy), visionexemplarLocateAnything 3BNVIDIA, visionproduced byMM-Grounding-DINOOpenMMLab, visionproduced byODISENVIDIA, visionproduced byOmDet-TurboZhejiang University, visionproduced byWeDetectTencent, visionproduced byYOLO-WorldTencent, visionexemplarYOLOETsinghua University, visionexemplarPapers
- Grounded Language-Image Pre-trainingLiunian Harold Li et al., CVPR 2022 (per arXiv comment), 2021 founding
- YOLO-World: Real-Time Open-Vocabulary Object DetectionTianheng Cheng et al., arXiv preprint, 2024 state of the art
- Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object DetectionShilong Liu et al., arXiv preprint, 2023 state of the art
- Simple Open-Vocabulary Object Detection with Vision TransformersMatthias Minderer et al., ECCV 2022 (per arXiv comment), 2022 state of the art
- A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and FutureChaoyang Zhu et al., arXiv preprint, 2023 survey
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2303.05499 https://arxiv.org/abs/2303.05499
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Open-Vocabulary Object Detection and Grounding", https://aurion.semai.id/topics/open-vocabulary-detection-and-grounding/, records of October 2026.