Semantic, Instance and Panoptic Segmentation Architectures
Networks that label every pixel of an image by class, by object instance, or both.
Tier Grades
Tier A: Frontier AI labs
feasible
Big-tech labs set the architecture line, from Mask R-CNN to Mask2Former.
Tier B: Indonesian AI labs
feasible
Training these networks on COCO and ADE20K is within reach of an Indonesian lab.
Tier C: Institution units such as ITB
feasible
A 5-author team produced Mask2Former, a size a university unit can match.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Fine-tuning a released segmenter on client imagery fits a proof of concept.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeTraining compute is not in the sources read.
- DataMask2Former reports 57.8 PQ on COCO panoptic, 50.1 AP on COCO instance and 57.7 mIoU on ADE20K.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 5 authors on the Mask2Former paper (arXiv 2112.01527).
- Time horizonYear.
- Shelf-life signalsPromptable and open-vocabulary segmenters replace closed-set models, One model handles all three segmentation tasks
- Pipeline roleTier A sets architectures; Tier C trains them on local datasets; Tier D applies them to area, defect and land-cover measurement in pilots.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
Mask2Formervisionexemplar
SegFormervisionexemplar
DeepLabV3+visionexemplarMask R-CNNvisionexemplarSAM 3.1 (Object Multiplex)visionproduced by
EoMTvisionproduced byBGNetvisionproduced by
PP-MobileSegvisionproduced byVPDvisionproduced by
X-Decodervisionproduced by
OneFormervisionproduced by
SegNeXtvisionproduced by
FFNetvisionproduced by
Mask DINOvisionproduced by
PIDNetvisionproduced by
SparseInstvisionproduced byMaskFormervisionproduced by
K-Netvisionproduced bySegFormer-B0visionproduced bySegFormer-B5visionproduced by
Segmentervisionproduced byQueryInstvisionproduced by
STDCvisionproduced by
BCNetvisionproduced by+41
DDRNetvisionproduced by
SETRvisionproduced bySCNet (instance segmentation)visionproduced byDETR ResNet-50 panopticvisionproduced by
CondInstvisionproduced byBlendMaskvisionproduced byPointRendvisionproduced bySOLOvisionproduced byPanoptic-DeepLabvisionproduced by
CenterMaskvisionproduced by
OCRNetvisionproduced byHRNetV2visionproduced by
MaskTrack R-CNNvisionproduced byLR-ASPPvisionproduced by
YOLACTvisionproduced byTensorMaskvisionproduced by
Fast-SCNNvisionproduced by
Hybrid Task Cascade (HTC)visionproduced byPanoptic FPNvisionproduced by
CGNetvisionproduced by
Attention-based segmentation networks (MMSegmentation)visionproduced byBiSeNetvisionproduced by
UPerNetvisionproduced by
UNet++visionproduced byDeepLabV3visionproduced byICNetvisionproduced by
DUC (Dense Upsampling Convolution)visionproduced byPSPNetvisionproduced by
SegNetvisionproduced by
U-Netvisionproduced byU-Net (2D)visionproduced byDeepLabvisionproduced by
FCN (Fully Convolutional Network)visionproduced byMOSAIC (mobile segmentation)visionproduced by3D U-Net (TensorFlow volumetric models)visionproduced byAutoSeg-EdgeTPUvisionproduced byERFNetvisionproduced byFastSegvisionproduced by
DRN (Dilated Residual Networks)visionproduced by
Intel instance and semantic segmentationvisionproduced byCircularNetvisionproduced byLinks
R&D TopicSemantic, Instance and Panoptic Segmentation Architectures
ModelsMask R-CNNMeta, visionDeepLabV3+Google, visionSegFormerNVIDIA, visionMask2FormerMeta, vision+61 more
Mask R-CNNMeta, visionDeepLabV3+Google, visionSegFormerNVIDIA, visionMask2FormerMeta, vision+61 moreWho Runs Itnone recorded
Models 65
U-NetUniversity of Freiburg, visionproduced byAutoSeg-EdgeTPUGoogle, visionproduced byDeepLabGoogle, visionproduced byMaskTrack R-CNNByteDance, University of Illinois Urbana-Champaign, Adobe Inc., visionproduced byUNet++Arizona State University, visionproduced byX-DecoderUniversity of Wisconsin-Madison, UCLA, Microsoft Research, visionproduced by3D U-Net (TensorFlow volumetric models)Google, visionproduced byDETR ResNet-50 panopticMeta, visionproduced byHRNetV2Microsoft Research, visionproduced bySAM 3.1 (Object Multiplex)Meta, visionproduced bySegFormerNVIDIA, visionexemplarU-Net (2D)University of Freiburg, visionproduced by+53
Attention-based segmentation networks (MMSegmentation)Institute of Automation, Chinese Academy of Sciences, JD.com, visionproduced byBCNetHong Kong University of Science and Technology, Kuaishou, visionproduced byBiSeNetHuazhong University of Science and Technology, Megvii, visionproduced byCircularNetGoogle, visionproduced byDeepLabV3Google, visionproduced byDeepLabV3+Google, visionexemplarDRN (Dilated Residual Networks)Princeton University, Intel, visionproduced byFast-SCNNToshiba, University of Cambridge, visionproduced byHybrid Task Cascade (HTC)The Chinese University of Hong Kong, SenseTime, visionproduced byIntel instance and semantic segmentationIntel, visionproduced byK-NetNanyang Technological University, SenseTime, visionproduced byMask DINOIDEA Research (International Digital Economy Academy), visionproduced byMask R-CNNMeta, visionexemplarOCRNetMicrosoft Research, Institute of Computing Technology, Chinese Academy of Sciences, visionproduced byOneFormerUniversity of Oregon, University of Illinois Urbana-Champaign, visionproduced byPSPNetThe Chinese University of Hong Kong, SenseTime, visionproduced bySegFormer-B0NVIDIA, visionproduced bySegFormer-B5NVIDIA, visionproduced bySegNeXtTsinghua University, Nankai University, visionproduced bySOLOUniversity of Adelaide, ByteDance, visionproduced byUPerNetPeking University, Massachusetts Institute of Technology, visionproduced byBlendMaskUniversity of Adelaide, visionproduced byCenterMaskETRI (Electronics and Telecommunications Research Institute), visionproduced byCGNetInstitute of Computing Technology, Chinese Academy of Sciences, visionproduced byCondInstUniversity of Adelaide, visionproduced byDDRNetHarbin Institute of Technology, visionproduced byDUC (Dense Upsampling Convolution)TuSimple, visionproduced byEoMTEindhoven University of Technology, visionproduced byFCN (Fully Convolutional Network)UC Berkeley, visionproduced byFFNetQualcomm, visionproduced byICNetThe Chinese University of Hong Kong, visionproduced byLR-ASPPGoogle, visionproduced byMask2FormerMeta, visionexemplarMaskFormerMeta, visionproduced byMOSAIC (mobile segmentation)Google, visionproduced byPanoptic FPNMeta, visionproduced byPanoptic-DeepLabGoogle, visionproduced byPIDNetTexas A&M University, visionproduced byPointRendMeta, visionproduced byPP-MobileSegBaidu, visionproduced byQueryInstHuazhong University of Science and Technology, visionproduced bySegmenterINRIA, visionproduced bySegNetUniversity of Cambridge, visionproduced bySETRFudan University, visionproduced bySparseInstHuazhong University of Science and Technology, visionproduced bySTDCMeituan, visionproduced byTensorMaskMeta, visionproduced byVPDvisionproduced byYOLACTUC Davis, visionproduced byBGNetvisionproduced byERFNetvisionproduced byFastSegvisionproduced bySCNet (instance segmentation)visionproduced byPapers
- Mask R-CNNKaiming He et al., arXiv preprint, 2017 founding
- Panoptic SegmentationAlexander Kirillov et al., CVPR 2019 (per arXiv comment), 2018 founding
- OneFormer: One Transformer to Rule Universal Image SegmentationJitesh Jain et al., arXiv preprint, 2022 state of the art
- Per-Pixel Classification is Not All You Need for Semantic SegmentationBowen Cheng et al., NeurIPS 2021 (per arXiv comment), 2021 state of the art
- Masked-attention Mask Transformer for Universal Image SegmentationBowen Cheng et al., CVPR 2022 (per arXiv comment), 2021 state of the art
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2112.01527 https://arxiv.org/abs/2112.01527
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Semantic, Instance and Panoptic Segmentation Architectures", https://aurion.semai.id/topics/universal-image-segmentation/, records of October 2026.