Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation

The short version

The problem

A model trained with reinforcement learning can teach another model what changed, but comparing only its starting and ending versions hides how those changes unfolded. A student may therefore be trained using a change that was partly reversed along the way.

The idea

Waypoint-OPD checks an intermediate version of the teacher as well as its starting and final versions. For likely next-word choices at a student’s current point in a response, it checks whether the early and late changes agree; it gives less weight to the final change when they point in opposite directions.

What it got

Across math and code tests, it had the best average score among the compared distillation methods in all six combinations of task and student size. For math, it beat endpoint-only transfer by up to 2.18 percentage points.