Critic-Inheriting Refinement After Verified Self-Rewriting
Repeatedly rewriting an agent can leave its value model out of step; Terminal-RSI trains on verified rewrites, carries critic weights across the actor-only stage, then recalibrates the critic before joint refinement.
Read the paper ↗What it is
- Verified rewrite The agent's proposed solution becomes training data only after review and a fresh run pass verification and leakage checks.
- Critic handoff The value model stays unchanged while the actor learns from accepted rewrites, then carries into the next refinement stage.
- Phase-local calibration Before actor and value model learn together, the actor is held fixed while the value model adjusts to the current policy.
Why it matters
A rewrite can change the policy enough to make its value model stale. Preserving critic weights reuses learned structure; recalibrating before joint refinement avoids treating that inherited value model as if nothing changed.
Before we get into the training loop, here is the short version: Terminal-RSI lets an agent learn from solutions it has rewritten itself, but only after those solutions pass verification. It keeps the value model across the rewrite, then gives that model a short adjustment phase before training actor and critic together. The point is to reuse useful value estimates without assuming they still fit the changed policy.
- Reliable self-training: A model can produce a convincing solution that fails the protected checks. The canonical example is a rewritten runbook that looks right but does not execute successfully in the target environment. Terminal-RSI does not treat the proposal itself as a label: a reviewer gate and fresh execution decide whether it enters replay. Generalizing from that example, self-generated supervision is useful only when there is an external way to reject bad candidates. What exactly does “verified” screen out?
The acceptance path checks a successful source run, applies a binary schema, artifact, and compatibility review, then freshly executes the candidate under the target harness and checks for both success and leakage. Accepted replay omits private runbooks and reviewer dialogue. This is more than filtering for good prose; it ties the supervision to an executable result. But once accepted solutions train a changed actor, what should happen to the critic?
- Reuse without pretending nothing changed: Think of the critic as a notebook of learned estimates. During actor-only rewriting, Terminal-RSI leaves that notebook untouched, then carries it to the next refinement stage. But a new actor can visit different states or produce different behavior, so the notebook may need adjustment. The notebook analogy has a limit: critic parameters are learned function weights, not literal records of individual experiences.
Before formalizing the schedule, the mechanism is simple: first let the value model adapt while the actor is fixed; only then let both learn together. The method calls this phase-local critic calibration and specifies four critic-only updates before joint training—not four updates before every minibatch. The initial refinement also has a warm-up critic prepared with the initial rewrite actor frozen. Later refinement inherits the previous retained critic across another actor-only rewrite. Why not simply train actor and critic together through the entire recursion?
- Separate discovery from improvement: The canonical example is one full recursive step: the actor discovers candidate rewrites, accepted ones provide actor-only supervision, and policy-refine then updates actor and critic jointly after calibration. In ordinary engineering terms, it separates data collection and supervised adaptation from the phase that uses a value baseline to improve the policy. More generally, each phase has a clear parameter contract: the rewrite changes the actor while critic weights stay fixed; refinement calibrates the critic with the actor frozen, then updates both. Other modules cover reward calibration and turn-aware credit assignment; here the question is how the critic survives the transition. What does this staged handoff buy, and what does it leave unresolved?
My view is that the cleanest idea is also the modest one: inheritance is an initialization, not a promise of calibration. It can preserve useful learned return structure, but the calibration phase exists precisely because policy and reward changes can make the old value estimates imperfect. The tradeoff is extra training work and dependence on a trustworthy verification gate; the benefit is that each recursion can reuse a critic rather than throw it away.
Read the deep dive
Why People Wanted It
A rewritten agent can produce better solutions, but the rewrite changes the actor while leaving the value model behind. If the next policy-refinement phase starts with an unprepared critic, its value estimates may not fit the actor it is supposed to guide. So the real question is not only how to find better demonstrations: how can refinement reuse what the critic has learned without pretending that its calibration survives a policy change?
The Core Move
Separate discovery from refinement: train the actor alone on independently verified rewritten solutions, preserve the critic through that actor-only transition, then recalibrate the critic with the new actor frozen before updating them jointly.
There is no single defining equation for this handoff in the described method. The key constraint is expressed directly: during recursive self-rewrite, the critic weights do not change, and each policy-refine phase performs critic-only calibration before joint updates.
Why Rewriting Is a Verified Supervision Step
Now the version for readers who want more of the machinery: a rewrite counts as training material only after it passes a chain of gates, not merely because the current agent proposed it. Discovery starts from a run that passes the protected verifier. A reviewer sees the public task and runbook, applies a binary schema, artifact, and compatibility gate, and accepted candidates are freshly executed under the target harness with private execution guidance. The fresh execution must verify successfully and pass leakage screening.
Only accepted replay is used for actor-only supervision. The procedure removes private runbooks and reviewer dialogue from the replay, and samples from the accepted multiset while preserving multiplicities. Thus the rewrite stage is a filtered route from successful behavior to new actor training data—not a license to imitate any plausible-looking self-generated answer. What happens to the value model while this new actor is being learned?
Why the Critic Survives the Rewrite but Must Be Recalibrated
Here is the part that gets technical: the critic is deliberately held fixed during actor-only rewriting, then carried forward as an initialization rather than treated as already calibrated. The first refinement phase begins with a warm-up critic prepared while the initial rewrite actor is frozen. Later, the next refinement phase inherits the retained critic from the previous refinement, even though an intervening rewrite has changed the actor.
Before each joint refinement phase, the actor is frozen and the critic receives four critic-only calibration updates. Those updates are phase-local: they happen before joint training, not before every PPO minibatch. After calibration, policy and value learning proceed together. The handoff therefore preserves learned return structure while explicitly allowing calibration to catch up with the current actor. Why alternate these phases instead of just continuing joint training throughout?
Why Recursive Policy-Refine Is a Separate Phase
Now for the deeper training logic: recursive policy-refine (RPR) turns verified rewrite data into a joint actor–critic improvement stage. The actor supplies decisions, while the critic supplies values used in the refinement objective; the reward module and the method's turn-aware advantage estimator provide the training signals. The important boundary here is parameter ownership: RSR changes the actor with critic weights unchanged, whereas RPR first calibrates the critic alone and then updates actor and critic jointly.
That separation makes the recursion legible. Each cycle retains a refined policy, uses it to discover and verify rewritten solutions, trains the next actor on accepted replay, transfers the critic, and recalibrates before the next joint improvement. Reward allocation and turn-aware credit assignment are separate mechanisms, not part of this handoff story. The remaining question is what this careful transfer buys—and what it cannot guarantee.
Minimal Causal Code
This plain loop shows the parameter handoff. verify_rewrite stands for the discovery, review, fresh-execution, and leakage checks; actor_only_train consumes only accepted replay. The calibration loop is shown separately from joint updates.
actor, critic = initialize_actor(), prepare_warmup_critic()
for cycle in range(num_cycles):
candidates = discover_rewrites(actor)
accepted = [item for item in candidates if verify_rewrite(item)]
# RSR: actor changes; critic weights are preserved.
actor = actor_only_train(actor, accepted)
# RPR: adapt the inherited critic to this actor before joint training.
freeze(actor)
for _ in range(4):
critic = critic_only_update(critic, actor)
unfreeze(actor)
actor, critic = joint_policy_value_update(actor, critic)
What You Gain, and What You Lose
You gain a recursive training loop that turns verified rewrites into actor supervision while retaining critic weights across the actor-only stage. Phase-local calibration acknowledges that transfer is useful but not equivalent to remaining calibrated after policy and reward changes.
The cost is extra machinery and training phases: rewrites must pass verification and screening, and each refinement stage adds critic-only calibration before joint updates. Weight inheritance is not a guarantee that the critic remains accurate; it is an initialization followed by recalibration. The method also depends on the quality of the verification and replay gates: weak gates can admit bad supervision, while overly restrictive gates can leave little accepted replay.
Preserve the critic through the rewrite, but recalibrate it before asking it to guide the next joint improvement.