Recovering Reliable Supervision via Critique-Guided On-Policy Distillation

The short version

The problem

When a language model makes a mistake partway through a solution, a stronger model asked to continue may copy that mistake instead of correcting it. This makes the stronger model’s training guidance least trustworthy exactly where the student most needs help.

The idea

The paper has the stronger model point out the student’s errors and give short, actionable fixes. The student then rewrites from the last identified mistake using that feedback; the teacher evaluates this corrected continuation with the feedback visible, but the student is trained without it—like learning from a worked correction after the hints are removed.

What it got

On six math benchmarks, this approach improved average accuracy over standard on-policy distillation by 4.57 percentage points for the 1.7-billion-parameter student. It also improved results for the smaller student and on most individual benchmarks.