Spatially Equivariant Patch Representations

Why People Wanted It

A patch encoder can produce a feature at every location without being taught what those features should mean. That is convenient until the intermediate features drift or collapse: downstream tokenization then depends on representations with no direct structural constraint. The real question is not merely how to extract a grid of tokens, but how to make each patch feature carry useful structure without letting it solve the task by memorizing its absolute slot.

The Core Move

The representation model directly supervises every patch feature, hides absolute position from its encoder, and supplies learned position queries measured relative to the visible input's center. In short: supervise the patch grid, but make its positional cue move with the visible region.

For image $X$, let $R_\theta(X)_{i,j}$ denote the feature at patch position $(i,j)$. The representation is the full grid $R_\theta(X) \in \mathbb{R}^{H\times W\times D}$, not a single pooled token.

Why Direct Patch Supervision Addresses Unstable Latents

Now the version for readers who want more of the machinery: the design treats structural supervision as a property of the patch features themselves, rather than assuming intermediate tokens will acquire stable structure on their own. The setup explicitly distinguishes supervised features—such as a contrastive model's CLS token—from unregularized intermediate latents, such as non-CLS tokens.

The concern is concrete: unregularized latents can diverge or collapse. The cited examples include outlier patch features in DINOv2, which motivated register tokens, and degraded feature quality in DINOv3, which led to a distillation stage aligning features with an earlier checkpoint. Direct supervision of the full patch grid is the proposed response: each patch feature participates as a representation vector under the structural objective, rather than being left as an unconstrained byproduct.

This does not mean every feature is guaranteed stable under every training setup; the move is to put the patch features under explicit supervision. What structural signal can constrain them without making location itself an easy shortcut?

Why Center-Relative Queries Encourage Spatial Equivariance

Here is the part that gets technical: the encoder loses absolute position information, but the representation model still needs a way to organize features spatially. It cross-attends from encoder outputs to learned position embeddings indexed by each patch's offset from the center of the visible input.

For a visible region with center $c$, a patch at absolute grid coordinate $(i,j)$ is queried using its relative offset $(i,j)-c$. In words, the query says where this patch lies relative to the visible region's center, not which fixed image-grid slot it occupies. The encoder therefore cannot simply pass an absolute position code through, while the output remains a feature at each absolute patch position; features at those positions are then compared by dot product.

That combination is the constraint: remove absolute position from the encoder, restore a relative spatial cue through the query, and evaluate features at their actual patch locations. The intended result is spatial structure that tracks the visible input rather than a fixed coordinate convention. How much spatial organization can this create without encoding absolute coordinates?

Minimal Causal Code

This sketch makes the relative-query choice visible. It is not a full model: encode_without_absolute_positions and the learned query table are placeholders for the architecture's encoder and cross-attention.

import torch

# x: [batch, patches, channels]; center_offset: [patches, 2]
# Position queries depend on offsets from the visible-region center.
relative_queries = position_mlp(center_offset)
encoded = encode_without_absolute_positions(x)

# Cross-attention produces one supervised representation per patch.
patch_features = cross_attention(
    query=relative_queries,
    key=encoded,
    value=encoded,
)

# The training objective directly uses this full patch grid.
loss = representation_objective(patch_features)

What You Gain, and What You Lose

Directly supervising patch features gives the representation objective an explicit handle on the grid, addressing the risk that intermediate latents drift or collapse without structural supervision. Center-relative queries provide spatial information while withholding absolute position from the encoder, so the representation can express relations to the visible region without relying on fixed absolute coordinates.

The tradeoff is that the feature design is not position-free: spatial organization depends on learned relative queries and on the visible region's center. Nor does the described constraint alone establish that all patch features will be stable or equally useful. It replaces an unconstrained intermediate-token assumption with a particular spatial bias—one that may be helpful when relative layout matters, but is still a modeling choice.