Understanding LLM Parameter Update Sparsity through the Lens of Fisher

The short version

The problem

When language models are trained to follow instructions or improve at tasks, most parameters may change very little, while a small subset does much of the work. It is hard to know in advance which parameters matter, or why this pattern appears across different training methods.

The idea

The paper measures how much each parameter can change the model’s output probabilities, using a quantity called the diagonal model Fisher. It shows that parameters with low Fisher tend to have small expected training updates, and that likely next words often respond similarly to a parameter—for example, “tea” and “coffee” may be affected in similar ways—so that parameter barely changes the model’s choice between them.

What it got

In a math-reasoning experiment, training only 0.1% of parameters chosen by this measure retained 85.21% of the improvement from training all parameters. The measure also picked out many of the parameters with the largest gradients, supporting its use as an explanation of where learning concentrates.