GroundingPI: From Perception Features to Robot Actions

Why People Wanted It

A visual foundation can locate objects or understand an instruction, but that does not automatically make its representation useful for choosing continuous robot actions. The practical question is how to connect different perceptual backbones to an action learner without also changing the action architecture, data, or training budget. GroundingPI uses a shared backbone-to-action interface so that the foundation representation is the main variable under comparison.

The Core Move

Take features from multiple depths of one pretrained backbone, map them into a common token interface, and let a fixed action model attend to those features while denoising an action chunk.

For sampled backbone states $H_{\ell_j}$, the conditioning tokens are

$Z_j = \operatorname{LN}(H_{\ell_j})W_B + e_j, \qquad C_j = \operatorname{Resampler}_{64}(Z_j).$

Here normalization, a shared-width projection, a depth embedding, and a learned-query resampler turn each sampled state into a fixed-size set of conditions for the action model.

Why Features From Several Depths Can Matter

Now the version for readers who want more of the machinery: the interface samples eight hidden states at normalized depths so different backbones contribute features from comparable relative positions. If the backbone has $N_B$ transformer blocks, the selected indices are $\ell_j=\operatorname{round}(j(N_B-1)/7)$ for $j\in\{0,\ldots,7\}$. This is a depth-normalization rule, not a learned choice of whichever layers happen to work best.

Each state is normalized and mapped from the foundation width $D_B$ to the action width. The projector $W_B$ is shared across the eight depths of a given backbone, while the learned embedding $e_j$ tells the resampler which sampled depth it is processing. The same learned-query resampler is shared across depths and models. This separates two jobs: the projector reconciles feature width, while the resampler compresses each layer's token sequence into 64 condition tokens.

The action model receives an ordered set of layer-specific conditions rather than one pooled summary. What useful information is preserved by that arrangement depends on the backbone and the task; the interface makes that transfer comparable, not guaranteed.

Why a Shared Resampler Makes the Comparison Cleaner

Here is the part that gets technical: the resampler's shared learned queries produce the same number and width of condition tokens regardless of a foundation's native token count or hidden width. Thus the cross-attention blocks see a common-shaped input, while the backbone-specific projector handles the width mismatch.

The foundation runs once for the observation and instruction. Its eight conditions are cached and reused during action denoising. For video-generation backbones, feature extraction is held to a fixed state—including observed-frame construction, diffusion timestep, condition mask, noise realization, resolution, and frame count—so those choices do not become extra comparison variables.

The action expert has a fixed 16-block layout: eight cross-attention blocks alternate with eight action self-attention blocks. In order, those cross-attention blocks use $C_0$ through $C_7$; the intervening self-attention blocks process the action-side sequence. If the interface is held fixed, what does the action learner actually optimize?

Why the Action Interface Keeps the Foundation in the Loop

Now for the deeper control pathway: a flow-matching Action DiT takes the robot state, the cached perceptual conditions, and a noisy action chunk, then denoises that chunk into continuous controls. Layer-wise cross-attention injects a different sampled condition at each designated cross-attention block; the action-side self-attention blocks operate between those injections.

This avoids giving a foundation-specific action head or a separate raw-text or proprioceptive cross-attention bypass to selected models. Language, visual perception, and dynamics information reach the action expert through the evaluated foundation representations, while the downstream control architecture and compute schedule stay shared. The design therefore asks a focused transfer question: how useful is each foundation's representation to this common action learner?

Minimal Causal Code

This sketch makes the conditioning path visible; it is not a full flow-matching training implementation.

# One backbone pass supplies all sampled depths.
hidden_states = backbone(observation, instruction, output_hidden_states=True)
conditions = []
for depth_id, layer_index in enumerate(sampled_layer_indices):
    h = hidden_states[layer_index]
    z = layer_norm(h) @ backbone_projector
    z = z + depth_embedding[depth_id]
    conditions.append(shared_resampler(z))  # fixed count of condition tokens

# Reuse conditions while the action expert denoises a noisy action chunk.
action = noisy_action_chunk
for block_id, block in enumerate(action_dit_blocks):
    if block_id % 2 == 0:
        layer_id = block_id // 2
        action = block.cross_attention(action, conditions[layer_id])
    else:
        action = block.action_self_attention(action)
    action = block.feed_forward(action)

What You Gain, and What You Lose

The shared interface makes foundation comparisons more controlled: the action expert, robot data, action representation, optimization budget, and inference procedure can stay fixed while the pretrained foundation varies. Sampling several depths gives the action learner access to features across the backbone rather than only one chosen layer, and resampling standardizes the conditions passed downstream.

The tradeoff is that this is an interface for comparison, not proof that every foundation's information transfers equally well. A fixed action expert and shared resampler constrain what can be expressed; depth sampling and feature compression may discard useful detail. The setup isolates representation transfer, but does not make the result a claim about closed-loop robot performance or about every possible action architecture.

Quiz

What is the main purpose of GroundingPI's shared backbone-to-action interface in the manipulation comparison?

To let each foundation use its own specialized action head and training budget.

To replace continuous action generation with language-only predictions.

To keep the downstream action learner and conditioning path controlled while varying the pretrained foundation.

$ check