Why On-Policy Tokens Can Have Low Fisher

Why People Wanted It

On-policy training samples responses from the model itself. A parameter can change many logits and still barely change the probability of the response if it moves likely alternatives together. So the real question is not simply which parameters affect logits, but which parameters change the model’s relative preference among the tokens it is likely to choose.

The Core Move

Split sequence Fisher into token-level contributions, then express each contribution as probability-weighted differences between a parameter’s sensitivities for pairs of candidate-token logits.

For a response $y=(y_1,\ldots,y_T)$, its score for parameter $\theta_i$ is $h_i=\sum_t \xi_{t,i}$, where $\xi_{t,i}=\partial_i\log p_\theta(y_t\mid s_t)$ and $s_t$ is the prompt plus preceding generated tokens. The diagonal model Fisher is $F_i=\mathbb{E}[h_i^2]=\mathbb{E}[\sum_t f_i(s_t)]$, with the expectation over prompts and on-policy responses.

Quiz

Why can the diagonal sequence Fisher be written as a sum of token Fisher contributions for on-policy generation?

The sequence score is defined as the average rather than the sum of token scores.

The cross-token score terms vanish in expectation because each later token score has conditional mean zero given its history.

Every token score is zero for every sampled response.

The logits at different positions are independent.

$ check

Why Sequence Fisher Splits Across Tokens

Now the version for readers who want more of the machinery: the sequence-to-token identity depends on on-policy sampling, not merely on writing a sequence probability as a product.

At prefix $s_t$, the next-token score has conditional mean zero: $\mathbb{E}[\xi_{t,i}\mid s_t]=0$. For an earlier position $u<t$, its score is already determined by the history available at $s_t$. Conditioning on that history therefore makes the expected cross term vanish: $\mathbb{E}[\xi_{u,i}\xi_{t,i}]=0$. Expanding the squared sequence score leaves the sum of token score squares in expectation. The cancellation is in expectation, not for each individual response.

This gives $F_i=\mathbb{E}[\sum_t \xi_{t,i}^2]=\mathbb{E}[\sum_t f_i(s_t)]$, where $f_i(s)=\mathbb{E}_{a\sim p_\theta(\cdot\mid s)}[(\partial_i\log p_\theta(a\mid s))^2]$. Positions after termination can be assigned zero score. Why should each token contribution be small? The answer is in the logits’ pairwise differences.

Why Token Fisher Measures Pairwise Sensitivity Differences

Here is the part that gets technical: differentiating log-softmax turns each token score into a contrast against the probability-weighted average sensitivity.

Write $j_{i,a}(s)=\partial_i z_a(s)$ for the sensitivity of token $a$’s logit to parameter $\theta_i$, and let $\bar j_i(s)=\sum_b p_b j_{i,b}(s)$. Then $\partial_i\log p_a=j_{i,a}-\bar j_i$. In words, the parameter’s effect on a token’s log probability is its logit movement minus the average movement across the distribution.

Squaring and averaging over the sampled token gives $f_i(s)=\sum_a p_a(j_{i,a}-\bar j_i)^2$. Equivalently, $f_i(s)=\frac12\sum_{a,b}p_ap_b(j_{i,a}-j_{i,b})^2$. The second form makes the constraint explicit: a uniform shift of all logits contributes nothing, and equal sensitivities among likely alternatives contribute little. This is variance under the model’s token probabilities, not a measure of the absolute size of logit changes. Which candidates matter most to that variance?

Why Probability Mass Must Align with Sensitivity

Now the deeper interpretation: the same sensitivity pattern can produce different Fisher depending on where the model places probability mass.

In the pairwise expression, each pair’s squared sensitivity gap is weighted by $p_ap_b$. Thus pairs of likely tokens dominate; a large difference involving only very unlikely tokens can contribute little. If high-probability candidates have similar sensitivities, token Fisher is small even when the parameter affects their logits substantially. A parameter that raises both “tea” and “coffee” similarly after “She drank a cup of” changes their shared plausibility, but does little to their relative preference.

A uniform-over-vocabulary reference keeps the prefix and sensitivities fixed while replacing the model probabilities with $1/V$. Comparing the two pairwise quantities asks whether the model’s probability mass lands on more similarly affected tokens than a uniform draw would. It is a diagnostic for alignment, not a claim that every parameter or prefix must behave this way.

Minimal Causal Code

This loop computes one prefix’s token Fisher from logit sensitivities. It makes the probability-weighted centering visible; it is not an efficient way to obtain all parameter derivatives.

import torch

# logits: [vocabulary], sensitivities: [parameters, vocabulary]
probs = logits.softmax(dim=-1)
centered = sensitivities - (sensitivities * probs).sum(dim=-1, keepdim=True)
token_fisher = (probs * centered.square()).sum(dim=-1)

What You Gain, and What You Lose

You gain a local explanation for low on-policy Fisher: likely candidate tokens can share a parameter’s logit movement, leaving their relative probabilities almost unchanged. The decomposition also says exactly what is being measured—probability-weighted variation in logit sensitivity, accumulated along generated prefixes.

You do not get a guarantee that Fisher is small for every parameter, prompt, or training stage. The sequence decomposition relies on on-policy sampling and its conditional-mean cancellation; it does not transfer unchanged to arbitrary response distributions. Nor does low Fisher mean a parameter has no effect: a common shift across likely logits can be substantial while nearly invisible to the next-token distribution. For a separate question—using Fisher to locate likely update coordinates—see the module on Fisher bounds.

Quiz

A parameter changes the logits of several high-probability candidate tokens by nearly the same amount. What does the pairwise expression predict?

Small token Fisher, because likely-token sensitivity differences are small.

Large token Fisher, because any nonzero logit movement increases it equally.

Exactly zero sequence Fisher for every response and prompt.

A value determined only by the number of tokens in the vocabulary.

$ check