Generative Prediction Differences

Contrastive learning needs negative examples to estimate how two views relate; RePR uses changes in conditional generative likelihood to estimate their shared information and trains feature dot products to match it.

Read the paper ↗

The short version

What it is

Generative PMI estimate Compare how well a generative model predicts one masked view when it can use the other view, versus when it cannot. That prediction advantage estimates what the views share.
Representation matching Train the feature vectors for the two views so their dot product matches the generative model’s estimate of shared information.
Scale-weighted similarity Choose which corruption strengths count most when measuring shared information; stronger corruption removes fine detail and leaves more robust dependencies.

Why it matters

Contrastive learning uses sampled negative examples to shape view similarity, which can be challenging in strongly conditional settings. RePR instead uses generative prediction to estimate shared information directly, though likelihood estimation can be costly for large inputs.

Before getting into the machinery, here is the short version: RePR asks a generative model how much easier one masked view is to predict when it can see another. It uses that prediction advantage as a target for the similarity between their learned features. That gives representation learning a route to shared-information supervision without sampled contrastive negatives.

Negative examples: Contrastive learning uses InfoNCE, whose behavior depends on sampling negatives. In strongly conditional settings, that sampling can be challenging. RePR tries estimating the relationship between two views directly instead.
What similarity means: A feature dot product is not automatically meaningful. RePR gives it a target tied to how much information the views share, rather than assuming a useful structure will emerge just by scaling representation learning.
Which information counts: Not every shared detail is equally useful. RePR’s scale-weighted version can emphasize dependencies that survive stronger corruption, rather than treating fine details and robust patterns alike.

So how can a generative model tell us what two views share? The canonical example is two masked crops of one image: if seeing one crop helps predict the other, they share information beyond what either crop alone gives you. More generally, RePR makes two randomly masked views, then estimates their PMI from conditional generative likelihoods. PMI is a measure of how much more often two outcomes co-occur than expected under independence. That definition is useful, but the crop example is the operational intuition: ask how much the first view improves prediction of the second.

The representation model then turns each view into a feature vector. RePR trains the dot product of those vectors to match the generative PMI estimate. In plain terms, the more shared information the generator estimates, the more the feature similarity is pushed to reflect it. This resembles the connection between contrastive embedding dot products and PMI, but RePR gets its target from generative prediction rather than sampled negatives. The method’s central move is not “make views look alike”; it is “make feature similarity track estimated shared information.”

So the real question was never just whether two views are related — it was which parts of their relationship should count. Ordinary PMI treats shared information equally. RePR’s scale-weighted version lets corruption strength decide what remains relevant: stronger corruption progressively destroys fine-grained information, so the remaining predictable dependencies can define similarity. Think of it as asking whether two notes still contain the same tune after background details fade; the analogy breaks because corruption is a model-defined noise process, not human listening.

The method uses diffusion-model likelihood estimates in its treatment of these quantities, and the supplied pages describe timestep weighting as a way to emphasize selected information scales. The paper also addresses tractability with incremental prediction, but that is a separate module’s topic. Overall, I like the directness of giving feature similarity a predictive target; the cost is that estimating likelihoods can be expensive for large inputs. If prediction-based similarity is the goal, how much computation is worth spending to measure it?

Read the deep dive

Why People Wanted It

Contrastive representation learning connects the similarity of two embeddings to the pointwise mutual information (PMI) between their inputs. But its InfoNCE objective depends on sampled negatives, which can be difficult in strongly conditional settings. RePR asks whether a generative model can estimate that shared information directly, so representations can be supervised without contrastive negatives.

The Core Move

RePR estimates how much knowing one masked view improves the generative prediction of the other, then trains the views’ feature dot product to match that estimate. In the paper’s notation, the basic target is the PMI estimated from conditional generative likelihoods, and the representation objective matches $\langle R_\theta(x),R_\theta(y)\rangle$ to that estimate.

The paper pages state the relationship and refer to equations for its formal definition, but the supplied page text does not include the displayed equations themselves. So the math here is limited to the dot product relationship explicitly present in the text; no likelihood-ratio formula is reconstructed.

Quiz

What does RePR use as the target for the dot product between two view embeddings?

A reconstruction of the original unmasked input by the representation encoder.

The number of negative examples sampled for the pair.

A class label assigned to the pair of views.

An estimate of the views’ pointwise mutual information from a conditional generative model.

Why Prediction Improvement Estimates Shared Information

Now the version for readers who want more of the machinery: PMI asks how much more often two observations occur together than they would if independent. RePR’s generative route uses the conditional likelihood of one view given the other, alongside the likelihood without that conditioning, to estimate this dependence. The paper describes this as estimating PMI from generative-model log probabilities; it does not provide the displayed equation in the supplied text, so the relationship is stated here without adding a formula.

The conditional model is the key: if the second view becomes easier to predict when the first is available, that predictive advantage is evidence of information shared between them. The estimate is not an embedding similarity yet; it is a target that the representation model is asked to reproduce. Why prefer this target to contrastive estimation when negatives are hard to sample?

Why the Dot Product Becomes the Supervision Target

Here is the part that gets technical: contrastive learning connects embedding dot products to PMI, up to an unidentifiable scalar offset, while RePR directly supervises the dot product to equal the PMI estimate obtained from the conditional generative model. The paper’s displayed equations are absent from the supplied page text, so the exact loss expression and offset treatment are not specified here.

For a training sample, the method randomly masks data to make two corrupted views, embeds both with a representation encoder, and uses the generative PMI estimate as the target for their dot product. This is the central alignment: a prediction-based estimate of shared information becomes a feature-space similarity target. What happens when the method should emphasize some shared information scales more than others?

Why Timestep Weighting Changes Which Similarities Matter

Now for a deeper qualification: ordinary PMI treats shared information equally, but RePR introduces a scale-weighted PMI so the representation objective can emphasize selected corruption scales. The paper explains that stronger corruption progressively removes fine-grained information, leaving dependencies that remain predictive under that corruption; a weighting over scales determines which dependencies matter more. The exact displayed weighting equations are not present in the supplied text, so no specific formula or schedule is asserted here.

This is a choice about the meaning of similarity, not merely how to estimate it: views can count as similar because they share details, or because they preserve information under stronger corruption. The method uses diffusion-based estimates for these PMI quantities, but incremental prediction and bootstrapped estimation are separate topics. Which information scale should a representation preserve for its intended use?

Minimal Causal Code

The following loop makes the supervision flow visible. pmi_target stands for the conditional generative model’s PMI estimate; its calculation is intentionally left abstract because the supplied text does not include its displayed formula.

import torch
import torch.nn.functional as F

# x and y are two randomly masked views of one training sample.
rx = representation_model(x)
ry = representation_model(y)

# Estimated by a conditional generative model in the actual method.
pmi_target = estimate_pmi_with_generator(x, y)

# The feature dot product is trained to match the estimated shared information.
predicted_pmi = (rx * ry).sum(dim=-1)
loss = F.mse_loss(predicted_pmi, pmi_target)
loss.backward()
optimizer.step()
optimizer.zero_grad()

What You Gain, and What You Lose

RePR avoids relying on sampled contrastive negatives for this supervision and uses generative prediction to estimate shared information directly. The tradeoff is that the target depends on generative likelihood estimation; the paper notes that likelihood estimation can be expensive or intractable for large inputs. Its scale-weighted variant also makes similarity depend on which corruption scales receive emphasis, rather than treating every shared detail identically.

Core Takeaway

RePR turns a generative model’s prediction advantage into a target for how similar two representations should be.