Finetuning with Sampling: SFT Learns Better Than You Think

The short version

The problem

Teaching a language model a new skill from expert examples can make it forget things it already knew. Training methods that avoid forgetting often depend on the model producing successful answers itself, which is a problem when the new task is too hard for it yet.

The idea

The paper first has the model reshape expert answers into different answers that preserve the same useful reasoning or facts, but look more like answers the model would naturally produce. It does this through repeated propose-and-check sampling, then uses these examples with standard supervised training—like helping a student restate a worked solution in their own words before practicing it.

What it got

On chemistry questions, the method raised accuracy from 34.3% to 66.0%. It also retained prior abilities better than the compared training methods in many experiments.