GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives

Qize Yu, Lianrui Fan, Boyu Chen, Jiaqi Liang, Xini Ding, Yue Chen et al.

arXiv 2609.39601 64 pages Read the paper

The short version

The problem: Robots and self-driving systems need to know exactly which object to act on and where it is, especially when scenes are crowded or targets are tiny. General image-and-language models can understand a scene but may not pinpoint the right place reliably.
The idea: GroundingPI teaches one model to connect words directly to points and boxes in an image. It writes those locations as special number tokens in the same output format—for example, it can name a cup and give the box around it—so many visual tasks use a common interface. The model is trained in stages, then further improved using feedback that rewards complete, accurate locations and text matched to the right image regions.
What it got: Across 34 visual-location benchmarks, it averages 73.68%, ahead of the larger GPT-6 Astra at 71.54%. The paper also reports gains when using it to help train robot-manipulation and driving systems.

Modules from this paper

GroundingPI’s Iterative Data Engine

Grounding supervision can be incomplete or contradictory; GroundingPI combines complementary teachers field by field, checks each task’s required evidence, and iteratively refines unresolved annotations.

data-enginegroundingannotationhuman-in-the-loop

GroundingPI: From Perception Features to Robot Actions

Robot action learning needs perceptual features in a usable form; GroundingPI projects and resamples features from multiple backbone depths into a shared interface for a fixed action model.

roboticsrepresentation-learningvision-language-modelsaction-conditioning

GroundingPI: Staged Spatial Training and Reward-Based Refinement

General multimodal pretraining alone does not teach precise structured grounding; GroundingPI adds coordinate-alignment supervised fine-tuning, then uses group-relative rewards to improve grounding coverage, localization, and OCR text–region consistency.

GroundingPImultimodal trainingspatial groundingGRPO

A Shared Coordinate-Token Interface for Visual Grounding

Separate grounding tasks usually need separate output formats; GroundingPI uses one language-conditioned protocol that pairs semantic labels with quantized point or box coordinates.

groundingmultimodalcoordinatesautoregressive-models