Distilling Categorical Judgments into Generator Updates

Why People Wanted It

A scientific generator needs to move toward a goal, but a language-model judge may be better at saying which of two designs fits a current criterion than at assigning a well-calibrated numerical score. The real question was not how to turn every judgment into a number. It was how to extract useful direction from coarse judgments while declining to train on cases the judge cannot resolve—and to keep those judgments relevant as the generator changes.

The Core Move

Ask the judge to label a batch of the generator’s current samples positive, negative, or abstained under a temporary sub-goal; turn positive–negative comparisons into DPO preferences, using the current generator as the reference.

For a positive candidate $x^+$ and negative candidate $x^-$, the preference is $x^+ \succ x^-$. Abstained candidates supply no comparison.

Why Ordinal Labels Avoid a Calibration Trap

Here is the part that gets more technical than the intuition: the label is not a hidden score, and the update does not need one. Under the accepted current sub-goal, the judge assigns each sampled candidate a label in $\{+, -, \varnothing\}$. The positive and negative sets are $P_t$ and $N_t$; the induced comparison set is $D_t=P_t\times N_t$. Thus every positive is preferred to every negative in that batch, while abstained candidates are left out.

This construction makes a substantive assumption: the batch labels define coherent regions on opposite sides of the current criterion. It does not claim to recover calibrated utility values. The supplied procedure also describes two independently executed judgment passes with permuted order; only samples with stable positive or stable negative labels enter the preference sets. Unstable and abstained cases are not forced into training comparisons.

Why the Reference Moves with the Generator

Now the version for readers who want more of the machinery: at round $t$, DPO uses $p_{\theta_t}$ both as the sampled generator and as the reference distribution. The preference update minimizes the round-specific DPO objective, yielding new parameters $\theta_{t+1}$. The reference therefore anchors each step to the current generator, rather than to a fixed model from an earlier stage.

This is online preference distillation: resampling after an update means the next judgments are elicited on the generator’s changed sample region. In the flow setting, terminal log-densities are intractable, so optimization uses a denoising-loss surrogate. That is an implementation route for the update, not a claim that the surrogate makes the exact finite-sample objective disappear.

Why the Update Has a KL-Regularized Target

Here is the deeper interpretation, which depends on an explicit modeling assumption: suppose the induced preferences follow a Bradley–Terry model for a latent utility $f_{g_t}(x)$. For reference $p_{\theta_t}$ and DPO temperature parameter $\beta$, the corresponding population target has the form

$p_t^*(x) \propto p_{\theta_t}(x)\exp\!\left(\frac{f_{g_t}(x)}{\beta}\right).$

In words, the target reweights the current generator toward higher implicit utility, with the reference resisting movement too far from its current distribution. The utility is implicit: the procedure observes ordinal preferences and neither queries nor directly fits $f_{g_t}$. This is a population-level interpretation, not a promise that finite data, limited model capacity, finite optimization, and the denoising surrogate land exactly on that target. It also relies on comparison paths connecting the relevant support and on a finite normalizing constant. What can a single locally regularized update guarantee about the full route to the final goal? Nothing by itself; curriculum progress is a separate question.

Minimal Causal Code

This pseudocode makes the data flow visible; dpo_loss stands for the preference objective, and the flow implementation may use its denoising-loss surrogate.

for round_id in rounds:
    samples = generator.sample(batch_size)
    labels = judge(current_subgoal, samples)  # positive, negative, abstain

    positives = [x for x, y in zip(samples, labels) if y == "positive"]
    negatives = [x for x, y in zip(samples, labels) if y == "negative"]
    pairs = [(pos, neg) for pos in positives for neg in negatives]

    reference = generator.snapshot()
    loss = dpo_loss(generator, reference, pairs)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

The code omits the described repeated judgment pass and stable-label filtering to keep the core state update visible; in an implementation that uses those passes, construct the sets from stable labels before pairing.

What You Gain, and What You Lose

You gain a supervision channel that uses categorical judgments rather than relying on calibrated raw scores, and one pass over a batch can produce positive–negative comparisons instead of requiring exhaustive pairwise duels. Abstention preserves ambiguity instead of turning it into a forced label. The moving reference regularizes each update to the generator that produced the current samples.

You lose information that a trustworthy graded score might contain: these labels express ordinal distinctions, not calibrated magnitudes. Preferences are only as reliable as the judgments and current criterion, and ambiguous or unstable samples contribute no pair. The implicit-target account is conditional on its preference-model and support assumptions; the finite update can miss that target, and a sequence of such steps does not by itself establish that the generator reaches the final goal.

Quiz

Under one current sub-goal, the judge labels one sample positive, one negative, and one abstained. Which samples contribute to the induced preference set?

Only the positive sample is used, because negative labels are not informative.

The positive and negative samples form a preferred pair; the abstained sample contributes no comparison.

All three samples are ranked by assigning the abstained sample a middle score.

The abstained sample is treated as negative so the update has a complete pair.

$ check