Fisher Bounds for Locating Likely Parameter Updates

Why People Wanted It

Post-training objectives look different on the surface: supervised fine-tuning follows demonstrations, reinforcement learning follows rewards, and on-policy distillation follows a teacher. But each objective ultimately changes model parameters through gradients. The real question is whether there is a common way to locate coordinates whose expected gradients are small, without pretending the objectives use the same learning signal.

The useful distinction is between a shared geometric factor and an objective-specific signal. The diagonal model Fisher supplies the shared factor; each objective supplies its own scale. This is a statement about an expected coordinate gradient for a single step, not a guarantee that a parameter will never move.

The Core Move

Write each expected coordinate gradient as a model score multiplied by an objective-specific signal, then apply Cauchy–Schwarz. The resulting common bound is

$|g_i| \leq \sqrt{F_i}\,\sqrt{\mathbb{E}_{\pi_\theta}[a(x,y)^2]}.$

Here $g_i$ is the expected gradient for coordinate $i$, $F_i=\mathbb{E}_{\pi_\theta}[h_i(x,y)^2]$ is its diagonal model Fisher, $h_i=\partial_i\log\pi_\theta(y|x)$ is the coordinate score, and $a$ is the signal associated with the objective. In words: the expected gradient is limited by the square root of the coordinate’s Fisher times the square root of the signal’s squared scale.

Quiz

What shared quantities appear in the common bound on an expected coordinate gradient?

The square roots of diagonal Fisher and the objective-specific signal’s squared scale.

The Fisher alone, regardless of the objective’s learning signal.

The largest sampled gradient and the number of training steps.

$ check

Why the Score Makes a Shared Bound Possible

Here is the part that gets technical: the key representation is not that all objectives have the same gradient, but that each expected coordinate gradient can be written as $g_i=\mathbb{E}_{\pi_\theta}[a(x,y)h_i(x,y)]$. The score $h_i$ is the derivative of the log probability of the sampled response with respect to coordinate $i$; the signal $a$ varies by objective.

The score has zero expectation under the model distribution. This follows from differentiating the normalization of response probabilities, under the stated assumptions that probabilities are differentiable and differentiation can be interchanged with expectation. Therefore adding or subtracting a response-independent baseline $b(x)$ from the signal leaves the expected gradient unchanged: $\mathbb{E}_{\pi_\theta}[b(x)h_i(x,y)]=0$. Cauchy–Schwarz applied to the score-signal representation gives the bound, with $F_i=\mathbb{E}_{\pi_\theta}[h_i^2]$. The bound is common; the signal scale is not.

Why the Signal Scale Changes with the Objective

Now the objective-specific version: supervised fine-tuning uses $a=w_\nu-1$, where $w_\nu(x,y)=\nu(y|x)/\pi_\theta(y|x)$ compares demonstration probability with model probability. Its squared scale is $\mathbb{E}_{x,y\sim\pi_\theta}[(w_\nu-1)^2]$. In words, this scale measures the squared mismatch between the demonstration distribution and the model distribution.

On-policy reinforcement learning uses reward centered by its prompt-conditional mean: $a=r(x,y)-\bar r(x)$, with $\bar r(x)=\mathbb{E}_{\pi_\theta}[r|x]$. Its squared scale is $\mathbb{E}_x\operatorname{Var}_{\pi_\theta}(r|x)$. Centering removes the prompt-only baseline without changing the expected gradient; the scale is reward variation across responses to a prompt.

PG-style on-policy distillation uses the teacher’s sequence signal centered by its model-conditional mean, with squared scale given by the expected conditional variance of that signal. Full-vocabulary on-policy distillation instead uses the token-level signal $A_T(x,y)$ and squared scale $\mathbb{E}_{x,y\sim\pi_\theta}[A_T(x,y)^2]$. The common bound therefore does not erase the distinction between objectives: it puts their different signals beside the same Fisher factor.

Why Small Fisher Is a Sufficient, Not Universal, Explanation

Here is the deeper qualification: a small diagonal Fisher is sufficient for a small expected coordinate gradient when the objective’s signal scale is controlled. Fisher alone does not bound the gradient independently of that scale, and the claim concerns a single step under the stated differentiation conventions. It does not say that every realized gradient is small or that a coordinate can never change over training. What does the shared bound let you conclude when the objective-specific scale is large?

Minimal Causal Code

This toy loop makes the score-signal product visible for a sampled response. It estimates an expected coordinate gradient by averaging samples; the inequality is about that expectation, not each sample.

# model.log_prob(x, y) returns log pi_theta(y | x)
score_sum = 0.0
gradient_sum = 0.0
for x, y in sampled_responses:
    logp = model.log_prob(x, y)
    score_i = torch.autograd.grad(logp, parameter_i, retain_graph=True)[0]
    signal = objective_signal(x, y)  # objective-specific a(x, y)
    score_sum += score_i
    gradient_sum += signal * score_i

mean_score = score_sum / len(sampled_responses)
mean_gradient = gradient_sum / len(sampled_responses)
# Under the model distribution, the expected score is zero.
# The expected squared score is the diagonal Fisher estimate.

What You Gain, and What You Lose

You gain a shared way to reason about where expected gradients can be small across supervised fine-tuning, on-policy reinforcement learning, and on-policy distillation. The objective-specific factor stays explicit, so a low Fisher coordinate is informative when the signal scale is controlled.

You lose any objective-free guarantee: a large signal scale can loosen the bound. You also get a bound on expected coordinate gradients for a single step, not a claim about every sample, cumulative parameter movement, or an automatic rule for selecting trainable parameters.

Quiz

When does a small diagonal Fisher imply a small expected coordinate gradient under this result?

Always, including arbitrarily large signals and every individual sample.

When the objective-specific signal scale is controlled; the result concerns an expected gradient for a single step.

Only when all post-training objectives use identical signals.

$ check