Mining Environment-Level Recovery Skills

Why People Wanted It

A tool-using agent can get stuck on an unavailable service, a wrong identifier, or an unchecked write. More reasoning is not necessarily the missing ingredient: the agent may need to recognize the failure and change its next action.

The problem is that clean interactions do not expose these recovery decisions. But adding errors alone is not enough either. An injected error describes what the agent encountered, not whether the agent ultimately failed. So the real question was never just how to make tools unreliable. It was how to turn observed differences between successful and failed behavior into reusable recovery guidance.

The Core Move

Perturb executable environments with observed tool failures, sample attempts in those noisy environments, and contrast jointly labeled successes and failures on the same task to extract environment-specific recovery rules.

The noise has an explicit control: each tool-return position is independently selected for replacement with probability $p_{\mathrm{inj}}$. A selected response is replaced with a compatible observed failure; otherwise the true response is retained. Success and failure labels are assigned afterward, not inferred from whether noise was injected.

Quiz

How does Agent-Reflex inject grounded tool failures?

Use the observed error fraction as the task's failure label.

Independently select each tool return for possible replacement by a compatible observed error.

Inject errors only after a rollout has been labeled unsuccessful.

Replace every tool return in any task selected to fail.

$ check

Why the Noise Comes from Tool Use

Here is the part that gets technical: failure injection operates on tool returns, using error forms collected from actual tool interactions rather than arbitrary invented messages.

Collected errors are grouped into eight actionable families: unavailable servers, authentication or permission failures, timeouts, missing resources, invalid arguments, execution exceptions, rate limits, and network errors. The renderer selects failures compatible with the invoked tool and balances sampling across families so that rare families receive recovery examples too.

Let $r_t$ be the true return from tool $f_t$, and let $J(f_t,r_t)$ denote its replacement by a compatible mined failure. The response shown to the agent is:

$$b_t \sim \operatorname{Bernoulli}(p_{\mathrm{inj}}),\qquad \widetilde r_t = \begin{cases} J(f_t,r_t), & b_t=1,\\ r_t, & b_t=0. \end{cases}$$

The draws are independent across tool-return positions. Thus $p_{\mathrm{inj}}$ is a per-return replacement probability, not a per-task failure probability and not the observed fraction of error returns in the mining corpus. Replacing a response also does not, by itself, imply rolling back the tool's state transition.

Environment construction retains clean tasks with verifiable solutions and perturbed tasks with verifiable recovery. This is an eligibility check on the tasks, not a rule that pairs a successful clean rollout with a failed noisy rollout.

Why Success and Failure Must Share a Task

Now the deeper sampling detail: correct and erroneous trajectories are outcome-labeled samples from the noisy ecosystem, not labels attached in advance to clean and perturbed runs.

A capability-diverse pool of nine models spanning five families attempts every task under the same sampling protocol. An analyzer and the environment verifier jointly label each rollout; disagreements are discarded. A failed rollout carries an error diagnosis, its first erroneous turn, and an environment-state snapshot taken before that turn.

Let $C_q$ contain the agreed correct trajectories for task $q$, and let $F_q$ contain the agreed erroneous trajectories and their annotations. Candidate contrasts are drawn from:

$$\mathcal P_q = \{(q,\tau^+,\tau^-,e,t^\star,s^\star): \tau^+\in C_q,\; (\tau^-,e,t^\star,s^\star)\in F_q\}.$$

Here $e$ is the diagnosis, $t^\star$ the first erroneous turn, and $s^\star$ its pre-action state. Pair contributions are capped per task, and each environment draws pairs from at least five distinct tasks. A task without both kinds of labeled attempt cannot supply a contrast.

Sharing a task makes the behavioral comparison interpretable. It does not imply that the attempts share a model, identical injected errors, or identical intermediate states.

Why the Extracted Rule Belongs to an Environment

Here is the deepest layer: the object being extracted is a mapping from a recurring bad pattern to a corrective action, not merely a summary of one failed attempt.

An analyzer examines paired trajectories, identifies recurring patterns, and converts them into executable recovery rules. For example, the illustrated file-and-comment environment connects wrong identifiers and unchecked writes with checking identifiers and verifying writes.

Each environment stores a record $\zeta_m=(d_m,T_m,B_m,K_m)$: its description and business objects, tools, observed bad patterns, and extracted reflection skills. This context gives a rule concrete objects and actions to refer to. Reuse is therefore across tasks within the environment; applicability to other environments is not automatic.

Minimal Causal Code

This deliberately unoptimized PyTorch sketch separates response injection, outcome labeling, and same-task pairing. The injected-response probability and pair cap are inputs. The harness helpers handle executable tools, joint labeling, and rule extraction; the loop makes the relevant state and dataset updates explicit.

import torch


def rollout(agent, task, env, failure_bank, p_inj, max_turns):
    state = env.reset(task)
    history, trace = [], []
    for _ in range(max_turns):
        action = agent.next_action(task, history)
        if action.is_final:
            trace.append((state, action, None))
            break

        before = env.snapshot(state)
        state, true_return = env.execute(state, action)
        shown_return = true_return
        if torch.rand(()).item() < p_inj:
            shown_return = failure_bank.draw_compatible_balanced(
                action.tool_name
            )
        # Replacing the message does not roll back the tool state.
        trace.append((before, action, shown_return))
        history.append((action, shown_return))
    return trace, state


def mine(env, tasks, model_pool, failure_bank, p_inj,
         max_turns, pair_cap, analyzer, verifier):
    assert 0.0 <= p_inj <= 1.0
    pairs = []
    contributing_tasks = set()

    for task in tasks:  # Already checked for solution and recovery.
        correct, errors = [], []
        for agent in model_pool:
            trace, final_state = rollout(
                agent, task, env, failure_bank, p_inj, max_turns
            )
            # Returns None on disagreement; otherwise an agreed label.
            label = joint_label(
                analyzer, verifier, task, trace, final_state
            )
            if label is None:
                continue
            if label.correct:
                correct.append(trace)
            else:
                errors.append((trace, label.error,
                               label.first_error_turn,
                               label.pre_error_state))

        # Only same-task, outcome-labeled attempts enter this pool.
        candidates = [(task, good, *bad)
                      for good in correct for bad in errors]
        selected = candidates[:pair_cap]  # Illustrative capped selection.
        pairs.extend(selected)
        if selected:
            contributing_tasks.add(task.id)

    if len(contributing_tasks) < 5:
        return None  # Does not meet the environment coverage requirement.
    return analyzer.extract_rules(env.description, env.tools, pairs)

A successful attempt and a failed attempt both come from the noisy sampling process. The code does not treat a recovery-validated noisy attempt as a failed trajectory, and it does not manufacture a contrast when one outcome pool is empty.

What You Gain, and What You Lose

Observed failure forms give the perturbations practical grounding. Same-task contrasts give the analyzer a shared objective against which to compare behavior. Environment-specific rules can refer to actual tools and objects rather than saying only “try again.”

The tradeoff is specificity and dependence on the collection process. Constructed perturbations do not cover every deployment failure, and response replacement does not simulate every possible service-side malfunction. Joint labeling filters disagreement but does not make the remaining diagnoses infallible. Requiring both outcomes also limits which tasks contribute contrasts.

Finally, a recovery library is knowledge, not a guarantee of reflective behavior. Later training stages teach local repair, global recovery, and responses to user corrections; they are separate from mining the rules.

Quiz

Which pairing correctly supplies a contrast for recovery-skill extraction?

Any two attempts for which a recovery verifier confirms success.

An agreed successful attempt and an agreed failed attempt on the same task, sampled in the noisy environment.

A clean attempt labeled successful by default and a noisy attempt labeled failed by default.

A success and a failure from unrelated tasks, provided they use the same model.

$ check