Context-Faithful Trajectory Processing

Why People Wanted It

A long-horizon agent does not produce one neat transcript. It can call tools, delegate work, revise artifacts, compact its history, and branch into different contexts. If training reconstructs those histories afterward, the reconstructed input may not be the one that actually conditioned an output. If every branch is expanded and trained independently, shared tokens can be counted repeatedly.

So the real question was never just how to store a conversation. It was how to preserve what the policy actually saw and generated, assess the work that remains when an execution ends, and turn branching records into training targets without duplicating shared prefixes.

The Core Move

Record each request and response as it happened, organize records into a shared-prefix tree, then select provenance-valid targets while keeping their execution identity. The paper gives no single formula for this processing pipeline; its core is a sequence of explicit record, score, mask, and admission operations.

Quiz

Why does QwenGyre retain each output's original conditioning context?

So training can use the context that actually conditioned the output, even if later compaction removes it.

So every branch can be converted into a separate, fully expanded transcript immediately.

So tool results can be treated as policy-generated targets.

$ check

Why Exact Request Records Matter

Here is the deeper machinery behind the intuition: each model-call record preserves exact input and output token IDs, output behavior log-probabilities, and associated metadata. Tool-call payloads are cached by call ID and restored before matching later histories, while incoming IDs remain available to associate tool responses. This lets the harness retain its message interface while training uses the recorded tokens.

The key constraint is that an output keeps its original conditioning context even if later history compaction removes that context from the harness’s current view. For a new request, the serialized context is matched against existing paths; matched nodes are reused, only new input content is encoded, and the request becomes a new node. Divergent contexts become branches, while a request with no matching prefix attaches to the root. Outputs generated independently remain separate nodes.

Why Scoring Can Survive an Incomplete Run

Now the scoring layer goes beyond the basics: at execution closure, the committed policy-generation request record is fixed and available artifact state is assessed under the task’s scoring rule. A timeout does not automatically make work unusable. Completed work can receive a valid score if it is assessable; the score is not derived from elapsed time, trajectory length, or termination status.

An assessment carries its execution, artifact version, and available branch reference, so a later assessment does not overwrite the state associated with an earlier one. A critique used as later input remains context, but evaluator provenance excludes it from policy targets. If evaluation fails, it can be retried against the same preserved state within the execution’s time budget. No valid reward by the budget’s end means those trajectories are excluded; a valid zero remains trainable. A complete failure with no assessable work can instead receive a zero-reward dummy trajectory.

Why Admission Uses Provenance and Shared-Target Masks

Here is the part that gets technical: first mask tokens that were not sampled from the policy. Policy-produced tokens are eligible targets; system and user prompts, tool results, and assistant-role content injected by the harness remain conditioning context only.

Candidate paths are leaves plus their root-to-leaf prefixes. Since executions can yield widely varying numbers of candidates, admission caps the number selected per execution at the stated maximum, $J_{\max}$. Within a role class, a leaf is drawn with probability proportional to its count of currently unmasked trainable tokens. Role priority puts main-agent task-driving turns ahead of delegated sub-agent work, and task-driving turns ahead of context-managing summaries within an agent.

When a path is drawn, its targets are fixed to the still-unmasked tokens on that path, then the entire path is masked for subsequent draws. This is the deduplication rule: a shared prefix contributes targets once, without changing targets already fixed by an earlier draw. Admission stops when no eligible tokens remain or the cap is reached. The excerpt names the maximum but does not provide the role-priority equation, so the priority is described here in words rather than reconstructed as a formula.

Quiz

What happens when admission selects a trajectory whose path shares already-claimed policy targets?

It duplicates the shared targets so every leaf receives a complete copy.

It fixes only still-unmasked targets, then masks the full path so shared targets are not selected again.

It turns all context tokens on the shared path into trainable targets.

$ check

Why the Loss Is Normalized Per Execution

The expert-level constraint is that an execution’s admitted targets form one deduplicated set, even if those targets are distributed across several selected paths. For execution $i$, each trajectory $r$ receives its fixed target set $T_r$; on-tree masking makes those sets disjoint. The execution target set is their union, $T_i = \bigcup_{r \in R_i} T_r$, and its length is $L_i = |T_i| = \sum_{r \in R_i}|T_r|$.

In words, the denominator is the number of admitted, deduplicated trainable tokens for that execution. Each target therefore has coefficient $1/L_i$ in the execution’s normalized loss; importance weighting and clipping do not change that denominator. An execution with no admitted targets contributes zero loss. The batch then averages over the original rollout executions, so splitting one execution across more selected trajectories does not give it extra aggregate normalization weight. Reassigning a fixed target set among paths leaves its execution ratio, clipping decision, and loss unchanged; changing which targets are admitted can still change the result.

Minimal Causal Code

This plain loop shows the target-mask invariant: each newly admitted path claims only targets not already claimed. It is a minimal illustration of that rule, not a full implementation of role ranking or probabilistic sampling.

# Each leaf_path is a root-to-leaf sequence of node IDs.
# policy_targets[node] is the set of token positions emitted by the policy.
claimed = {node: set() for node in policy_targets}
admitted = []

for leaf_path in selected_leaf_paths:
    targets = {}
    for node in leaf_path:
        fresh = policy_targets[node] - claimed[node]
        if fresh:
            targets[node] = fresh

    if not targets:
        continue

    admitted.append((leaf_path, targets))

    # Fix this trajectory's targets, then mask its whole path for later draws.
    for node in leaf_path:
        claimed[node].update(policy_targets[node])

What You Gain, and What You Lose

You gain faithful conditioning contexts despite later compaction, shared-prefix storage instead of immediate full-history expansion, and a way to learn from assessable partial work. Provenance masks keep injected context out of policy targets; path masking prevents shared targets from contributing twice; execution-level normalization keeps an execution’s aggregate weight from depending on how many paths carry its targets.

The costs are real: request recording and payload restoration add bookkeeping; tree matching and deferred materialization make admission more involved than treating every transcript as an independent sample; and unresolved scoring can exclude otherwise recorded trajectories. A role-prioritized cap also deliberately leaves some candidate paths out. This is a data-integrity design, not a guarantee that every branch or every unfinished run becomes training signal.