Self-Context Distillation
Guided attempts can contain useful actions that rely on hints the trained model will not see; self-context distillation selects supported actions and trains on them without the guidance.
Read the paper ↗What it is
- Self-context distillation Train a model on selected responses from its own helped attempts, but only after removing the help and checking that each retained action still fits what the model can see.
- Context-supported supervision A correct action is not enough: its reasoning must be supported by the remaining observations, not by a hint removed from the training context.
Why it matters
This turns guided attempts into usable action examples while reducing the risk that training teaches a model to rely on information it will not receive.
Before getting into the mechanics, here is the short version: Nemotron tried tasks with extra help, then learned from selected parts of those attempts after the help was removed. The important catch is that an action only makes a good example if the model could have chosen it from the context it still sees.
-
Useful attempts: A model’s own run can contain behavior worth copying, but a run is not automatically a good demonstration just because it completed. The recipe considers outcomes, completion, grading, trajectory length, and comparisons with earlier attempts on the same task. The canonical example is a task where an earlier attempt failed and a later guided attempt does better: comparing the two can help identify a more useful trajectory. That comparison helps choose runs; it does not by itself certify every decision inside one. So what makes a particular action safe to learn from?
-
Help changes the context: During collection, Nemotron could receive terminal-use guidance and, for selected previously failed tasks, a short hint aimed at the observed failure. Think of a hint like a note handed to someone midway through a job: it may point them toward the right fix, but deleting the note later does not erase what they learned from it. The analogy has a limit: the training question is about whether the recorded decision is supported by its visible context, not whether the model literally remembers a note. So how do you avoid teaching an action whose explanation vanished with the hint?
-
Supported actions: The key move is to reconstruct each selected attempt without its guidance, then keep only decisions whose reasoning is still supported by the observations left in view. For example, if a hint supplies the crucial fact behind a next action, that action should be excluded when the hint is removed—even if it happened to work. The remaining model responses become training targets, with preceding observations providing context. In other words, teach the model what it did when the visible situation could justify doing it; do not smuggle the removed hint into the example. Which useful decisions will this rule discard along with the unsupported ones?
The resulting SCD dataset contained 3,365 training examples with approximately 7.44 million supervised tokens. In the reported comparisons, SCD and selected teacher decisions each reached 0.31, but their datasets and training configurations differed, so that is not a matched comparison. Overall, I like the precision of the context check: it treats a demonstration as more than a successful answer. Its cost is that deciding what a context really supports takes care; deleting guidance is not a mechanical cleanup step.
Read the deep dive
Why People Wanted It
A capable agent can solve a task with help it will not have at deployment: terminal-use guidance, or a hint aimed at a failure it made before. Those guided runs may contain useful behavior, but copying them wholesale creates a mismatch: the model is trained to reproduce decisions whose supporting information has disappeared. So the real question was not merely how to collect successful attempts. It was how to keep the useful decisions while ensuring each one still makes sense from the context the model will actually receive.
The Core Move
Self-context distillation (SCD) uses a model’s guided attempts as candidate demonstrations, removes the guidance, and supervises only selected actions whose reasoning remains supported by the visible context. No mathematical formula is given for SCD here; its core is a data-selection and context-checking procedure.
Why Guidance Removal Is More Than Deleting Text
Here is the part that gets technical: removing a hint can invalidate a later action even when the action itself is correct. A hint might supply a fact that the model then uses in its reasoning; if that fact is absent from the reconstructed training context, the decision is no longer justified by what the model can see. SCD therefore reconstructs the context without terminal guidance or hints and excludes steps whose reasoning depends on information supplied only by a removed hint.
This is a support check, not just text cleanup. The retained responses are paired with their preceding observations, so supervision teaches actions in the context that remains. The key constraint is that the action’s rationale must be available from that context, rather than smuggled in by collection-time help. What counts as enough support for a decision in a long trajectory?
Why Attempt Selection Changes the Training Signal
Now the version for readers who want more of the machinery: a guided run is only a candidate, not automatically a training example. The recipe filters attempts using outcome, completion, grading, and trajectory length, and compares them with earlier attempts on the same task. Those checks select useful trajectories before the context-support check decides which individual steps can survive guidance removal.
The distinction matters: attempt-level selection asks whether a trajectory is worth learning from; step-level exclusion asks whether a particular decision still has visible support. The resulting supervision consists of selected model responses, conditioned on preceding observations. The final SCD dataset contained 3,365 training examples and approximately 7.44 million supervised tokens. How should you interpret a useful attempt when its best action is supported by a mix of visible context and removed hints?
Minimal Causal Code
This plain loop makes the context check visible. supported stands for the explicit review that a decision does not rely on removed guidance; it is not a built-in test supplied by this sketch.
training_examples = []
for attempt in guided_attempts:
if not passes_attempt_filters(attempt):
continue
context = reconstruct_without_guidance(attempt)
for step in attempt.steps:
if depends_on_removed_hint(step):
continue
if not supported_by_visible_context(step, context):
continue
training_examples.append({
"context": context.before(step),
"response": step.model_response,
})
What You Gain, and What You Lose
You gain a way to turn a model’s own helped attempts into action supervision without teaching it to depend on information removed from its training context. The retained responses are grounded in preceding observations, and the selection process can use outcomes and comparisons with earlier attempts.
You lose coverage whenever a potentially useful decision depends on a hint that cannot be justified from the remaining context. The method also depends on making careful attempt selections and reliably identifying which reasoning relies on removed information. In the reported comparison, selected teacher decisions and Nemotron’s guided attempts each reached a score of 0.31, but the datasets and training configurations differed, so that comparison does not establish which approach is more effective under a matched budget.
A guided action is worth distilling only when its support survives the removal of guidance.