Agent-Reflex: Verified Training for Recovery

Why People Wanted It

An agent can fail in more than one way. It may make a local mistake while using a tool, commit to a bad plan that needs a fresh start, or drift from what the user meant and need to respond to a correction. Ordinary successful demonstrations mostly teach what a good trajectory looks like; they do not necessarily teach what to do once the agent is already off course. So the real question was never just how to collect more examples. It was how to turn recovery into behavior the policy can learn, while keeping training inputs close to what the agent could actually see at deployment.

The Core Move

Reflective Supervised Fine-Tuning (ReSFT) creates three kinds of recovery continuations, accepts only episodes the environment verifier marks successful, and trains on the visible context without exposing privileged recovery skills. The distinction is where recovery begins: the error state, the original task state, or a user-correction turn.

Why the Restart Point Changes the Skill

Here is the part that gets technical: SR and KR differ not just in wording but in the state from which the continuation is generated. For an erroneous trajectory, let $t_\star$ be its first wrong turn, $s_{t_\star}$ the environment snapshot before that turn, and $h_{t_\star}$ the visible history. Let $\kappa$ be the matched recovery skill. A continuation is retained only if the verifier says it completes the task from the restored state.

In Self-Reflection Sampling (SR), restore $s_{t_\star}$ and continue from the history with the skill appended as temporary guidance. In compact form, $y_{SR} \sim \pi_0(h_{t_\star} \oplus \kappa; s_{t_\star})$, with the requirement that $\mathrm{Ver}_{s_{t_\star}}(h_{t_\star} \oplus y_{SR})=1$. This says: generate a repair from the point just before the first error, then keep it only when the environment accepts the completed episode. The skill helps elicit the continuation but is removed from the stored input, so the learned example asks the policy to recover without a hidden skill prompt.

KR targets a different failure: the initial plan itself may be wrong, so repairing the next tool call is not enough. Skill Recovery Sampling restores the initial state $s_0$ and restarts with the original task and temporary skill guidance: $y_{KR} \sim \pi_0(q \oplus \kappa; s_0)$, again retaining only verifier-accepted completions. In words, the agent gets another attempt from the beginning, rather than being trained to patch a plan whose premise has failed. Its skill text is also removed before storage. The question this leaves open is how to teach recovery when the missing information comes from the user, not the environment?

Why User Corrections Stay Visible

Now the version for readers who want more of the machinery: UR turns the matched skill into a natural-language user message, rather than placing the skill itself in the agent's context. A simulator maps the skill and error-time context to a correction, $u_\kappa=U(\kappa,h_{t_\star},s_{t_\star})$. This means the policy receives an ordinary conversational intervention; the privileged skill is not passed through as a prompt.

The environment is restored to $s_{t_\star}$, the correction is inserted after the erroneous turn, and the continuation is sampled from that corrected interaction. Unlike SR and KR, the synthesized message remains in the stored input because a user turn is deployment-visible. The correction is not automatically a command to obey: the simulator can produce direct corrections, partial hints, ambiguity, or deliberately incorrect advice. That setup makes the target behavior replanning in context, not blind compliance. How can these distinct recovery targets become plain training examples without leaking the machinery that generated them?

Why the Stored Dataset Is Deployment-Shaped

The training merge combines successful trajectories from the starting policy with the verifier-accepted SR, KR, and UR continuations. The crucial constraint is what the policy sees: simulator snapshots restore environment state but are never serialized as context; SR and KR skill text is removed; UR retains the rewritten user message but not the skill prompt. Thus the stored examples preserve the useful visible interaction while withholding privileged state and skill instructions.

The continuation actor is Qwen3-8B, matching the trained backbone, and the UR user simulator is GPT-4.1. A sample enters the corpus only when the environment verifier accepts the episode. The resulting corpus contains 54K trajectories, combining verified starting-policy successes with reflective corrections across the three sampling strategies. These details define the data-generation setup; the learning target remains the visible recovery behavior. What does this carefully shaped supervision buy, and what does it leave unsolved?

Quiz

Which pairing correctly matches each ReSFT sampling strategy to its recovery context?

All three resume from the initial state, differing only in whether the skill is stored.

SR resumes before the first error; KR restarts from the original task; UR inserts a user correction and continues from the error-time state.

SR restarts from the original task; KR resumes before the first error; UR hides the correction and continues from the initial state.

SR and KR keep the skill in the stored context, while UR stores the simulator snapshot.

$ check

Minimal Causal Code

This sketch makes the three restore points and the verifier gate visible. The rollout and environment interfaces are schematic; the key is that temporary skill guidance is used to generate candidates but is not automatically copied into stored SR/KR context.

def make_recovery_example(env, policy, verifier, task, history, skill, kind, user_sim=None):
    if kind == "SR":
        snapshot = env.snapshot_before_first_error()
        visible_context = history
        env.restore(snapshot)
        generation_context = visible_context + [skill]
        continuation = policy.generate(generation_context, env)
        stored_context = visible_context

    elif kind == "KR":
        env.reset()
        generation_context = [task, skill]
        continuation = policy.generate(generation_context, env)
        stored_context = [task]

    elif kind == "UR":
        snapshot = env.snapshot_before_first_error()
        env.restore(snapshot)
        correction = user_sim.rewrite(skill, history, snapshot)
        visible_context = history + [correction]
        continuation = policy.generate(visible_context, env)
        stored_context = visible_context

    else:
        raise ValueError("unknown recovery type")

    episode = env.finish_episode(continuation)
    if verifier.accepts(episode):
        return {"context": stored_context, "continuation": continuation}
    return None

What You Gain, and What You Lose

You gain supervised examples that distinguish three recovery situations instead of treating every failure as the same: repair from just before an error, restart the original task when the plan is flawed, and replan after a visible user correction. Verifier acceptance filters out continuations that do not complete the task, while removing privileged state and skill text from stored contexts keeps the training input closer to deployment.

You also inherit the limits of generated supervision. Recovery examples depend on the continuation actor, the quality of the matched skill, the simulator's user messages, and the environment verifier's judgment. A verifier can establish task completion according to the environment; it does not by itself guarantee that a recovery explanation is robust or that an ambiguous user correction was interpreted well. ReSFT teaches from accepted examples; it does not eliminate the need to test recovery under new errors and interactions. The next stage, on-policy optimization, is a separate module rather than part of this data-construction story.