Iterative Feature Prediction
Pixel targets can limit representation learning; RePR repeats training with patch features from an earlier model as prediction targets.
Read the paper ↗What it is
- Earlier-model targets After one training cycle, use its image features as the targets for a later cycle instead of asking the later cycle to predict pixel patches.
- Matching locations The feature target comes from the same image location as the patch target it replaces.
Why it matters
This lets RePR repeat its training approach using learned features as prediction targets. The described feature calculation can be shared across repeated views, making this reuse implementationally cheap.
Before we get into the mechanics, here is the short version: RePR can run its training idea again, but change what the next run predicts. Instead of targeting image patches directly, it uses features made by a representation model from an earlier cycle. The hope is that learned features make useful targets for building a stronger representation.
-
A different target: Pixel prediction asks the model to account for image content directly. In RePR's iterative version, the canonical concrete example is one image patch: the later cycle predicts the feature attached to that same location by the earlier model. The location stays fixed; the thing being predicted changes from a pixel patch to a learned feature. This is not a claim that the feature contains every detail of the original patch.
-
A recursive training move: The previous model first embeds the whole image. Then its feature at a patch location replaces the incremental patch target for that location in the next cycle. Think of it as carrying forward a notebook from the last trip and asking the next traveler to predict the notes for each matching place. The analogy stops there: the feature is a learned representation, not a literal record of everything seen.
So why should a feature target help rather than simply discard information? The section motivates the move by pointing to representation-learning work where feature-space self-supervision improved representation quality over direct pixel prediction. It also states a specific optimality connection: unweighted PMI estimation for latent prediction matches pixel prediction when the representation factorizes PMI. In words, under that condition, changing from pixels to latent features does not change the PMI being estimated. With timestep weighting and bootstrapped prediction, feature-space prediction gives further control over the learning signal; those are separate mechanisms, not the focus here.
- Reuse rather than recompute: The earlier model embeds the entire image, and that feature calculation can be reused across repeated views. That is the practical detail: the target comes from a previous model, but calculating it for each repeated view need not mean repeating the whole embedding computation. The stated implementation calls the view count . In words, this is the number of repeated views for which feature calculation is shared.
So the real question is not whether the target is still a pixel patch—it is not—but whether recursively learned feature targets yield useful additional gains. The described result is that they can; it is not a promise that every dataset or setup improves.
Read the deep dive
Why People Wanted It
Pixel prediction gives a model a direct reconstruction target, but a representation learner can instead be asked to predict features produced by an earlier representation model. The practical question is whether replacing the incremental patch targets with learned features from a previous training cycle can yield stronger representations. RePR explores that change recursively: train once, use the resulting representation to define targets, and train again.
The Core Move
Repeat the training technique, replacing predicted incremental patches with features from a previous cycle. For an image , the earlier representation model embeds the image, and the target at a patch location is the feature for that same location.
Why the Earlier Model Supplies the Target
Here is the part that gets technical: the earlier model is used to embed the entire image, and the incremental patch target is replaced by the representation at the corresponding image location. This is not a new target-selection rule based on a different patch location; the described substitution is location-matched.
The section states that an unweighted PMI factorization makes latent prediction equivalent to pixel prediction at optimality, expressing this as . In words, under that stated optimality condition, predicting the representation has the same PMI as predicting the pixel input. With timestep weighting and bootstrapped prediction, feature-space prediction offers further control over the learning signal; those mechanisms are covered elsewhere.
Why Reusing Views Makes the Recursion Practical
Now the implementation detail beyond the basics: RePR shares feature calculation across the repeated views. The earlier representation model embeds the whole image, and that embedding can be reused across views, making the feature calculation implementationally cheap. This is a reuse strategy, not a claim that the representation model is trained for free.
Minimal Causal Code
The paper's description gives the target substitution and feature reuse, but no standalone formula for an iterative training loop. This pseudocode makes the causal order visible: first obtain an earlier model, then use its same-location features as targets for a later cycle.
# Conceptual PyTorch-style sketch; the target substitution is the point.
old_model = train_cycle(initial_model, image_targets=True)
for batch in loader:
with torch.no_grad():
old_features = old_model.embed_entire_image(batch)
prediction = prediction_model(batch)
targets = patch_features_at_matching_locations(old_features)
loss = incremental_patch_loss(prediction, targets)
loss.backward()
optimizer.step()
optimizer.zero_grad()
What You Gain, and What You Lose
The reported motivation is stronger representation quality from feature-space self-supervision, and the described method says recursively applying the training technique can yield additional gains. The target is no longer the original pixel patch: it is a feature from an earlier model, tied to the same image location. That makes the target depend on a previous representation cycle, and the implementation shares the earlier model's feature calculation across repeated views. The section does not specify a universal gain or a numerical improvement, so treat the benefit as a reported possibility, not a guaranteed outcome.
RePR's iteration changes the target: a later cycle predicts same-location patch features made by an earlier representation model.