Retrospective Lessons and Action Replay

Why People Wanted It

A capable agent can finish a task and still fail to learn the lesson that would help on the next one. Reviewing a past attempt can produce useful advice—fix the runtime bottleneck before changing the model, for example—but advice alone is not an example of how to act. The real question is whether an agent can learn from its own retrospective lessons without losing the concrete reasoning and actions that made its earlier behavior trainable.

The Core Move

Train on self-written reviews of past attempts while replaying examples of the reasoning and actions those attempts contained. The training mixture has two targets: a retrospective response conditioned on a transcript, and the original reasoning-plus-next-action target conditioned on its context.

Why a Lesson Needs a Transcript

Now the version for readers who want more of the machinery: a reflection example is not just a free-standing rule; its transcript is the context that makes the retrospective response about a specific run. Let a run be $\tau=(o_1,a_1,\ldots,o_T,a_T)$, with observations $o_t$ and agent responses $a_t$. The reflection target $r$ is trained conditionally as $p(r\mid\tau)$: the model sees what happened, then learns to produce the lesson about that attempt.

This pairing matters because advice such as “fix runtime first” is only useful when grounded in the actual failure and subsequent change. In the reported process, the model reviewed its runs, lessons were checked for format and consistency with the run, and an LLM judge rated them. The lesson is therefore supervision about interpreting a trajectory, not a replacement for the trajectory itself.

Why Replay Keeps Behavior in the Mix

Here is the part that gets technical: the mixture assigns different supervised targets to the two example types. For reflection examples, the transcript supplies context and the retrospective response is the target; for replay examples, the original reasoning and next action are the target. In shorthand, the objective can be written as $\mathcal{L}=\mathcal{L}_{reflect}+\mathcal{L}_{replay}$, where the terms denote losses over their respective examples—not a claim that their weights or contributions are equal.

The reported recipe combined 972 reflection examples with 500 replayed action examples from the first-stage SCD dataset, starting from the SCD checkpoint. It trained for one epoch at a learning rate of $3\times10^{-5}$ over 184 steps. Reflection-only variants regressed relative to their SCD parent in the diagnostic evaluation; the combined recipe did better than those variants and had the highest score in the reported 22-task comparison. That comparison does not isolate the ingredients: learning rate and mixture changed together, and there was no matched replay-only control. So what can the result tell us about the value of reflection by itself?

Minimal Causal Code

This sketch makes the distinct targets visible. It leaves batching, masking, and loss weighting to the surrounding training code.

for example in examples:
    if example.kind == "reflection":
        context = example.transcript
        target = example.retrospective_response
    else:  # replay
        context = example.action_context
        target = example.reasoning_and_next_action

    logits = model(context)
    loss = token_loss(logits, target)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

What You Gain, and What You Lose

The gain is a training signal for turning an agent's own attempt into a reusable lesson while retaining examples of concrete agent behavior in the mixture. Reflection is incorporated during training; evaluation uses the same Terminus-2 harness without an additional reflection loop.

The tradeoff is attribution. The combined recipe performed better than reflection-only variants in the reported comparison, but the changing learning rate and mixture and the missing matched replay-only control leave the individual contribution of reflection unresolved. This is a recipe-level result, not proof that either ingredient alone caused the improvement.