Robot Perception for Manipulation and Navigation
Models that give a robot the 3D facts it needs to act: object pose, depth, touch and its own location.
Tier Grades
Tier A: Frontier AI labs
feasible
NVIDIA trains FoundationPose on large-scale synthetic data.
Tier B: Indonesian AI labs
feasible
FoundationPose runs on a novel object from its CAD model without fine-tuning, so use needs little compute.
Tier C: Institution units such as ITB
feasible
Applying FoundationPose to new objects needs only a CAD model or a few reference images.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Pose estimation from a CAD model without fine-tuning fits a proof of concept for robotic picking or inspection.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeFoundationPose: training compute not stated in the abstract; trained at large scale on synthetic data.
- DataFoundationPose: large-scale synthetic training aided by a large language model; works on novel objects from a CAD model or a few reference images.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 4 authors on the FoundationPose paper (arXiv 2312.08344).
- Time horizonMonths.
- Shelf-life signalsFoundationPose reports beating methods specialised for each setup, which retires per-object pose models.
- Pipeline roleTier A releases perception models; Tier B and Tier C test them on local objects; Tier D builds picking or inspection proofs of concept on them.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Links
R&D TopicRobot Perception for Manipulation and Navigation
Models
FoundationPoseNVIDIA, robot policyCenterPoseNVIDIA, visionESS and Bi3D (Isaac)NVIDIA, robot policy
SparshMeta, robot policy+1 more
FoundationPoseNVIDIA, robot policyCenterPoseNVIDIA, visionESS and Bi3D (Isaac)NVIDIA, robot policySparshMeta, robot policy+1 moreUse Casesnone recorded
Who Runs Itnone recorded
Papers
- Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp MetricsJeffrey Mahler et al., arXiv preprint, 2017 founding
- ViNT: A Foundation Model for Visual NavigationDhruv Shah et al., CoRL 2023 (per arXiv comment), 2023 state of the art
- AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal DomainsHao-Shu Fang et al., arXiv preprint, 2022 state of the art
- ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAMCarlos Campos et al., arXiv preprint, 2020 state of the art
- Habitat: A Platform for Embodied AI ResearchManolis Savva et al., ICCV 2019 (per arXiv comment), 2019 state of the art
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2312.08344 https://arxiv.org/abs/2312.08344
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Robot Perception for Manipulation and Navigation", https://aurion.semai.id/topics/robot-perception-for-manipulation-and-navigation/, records of October 2026.