Collecting Verified Long-Horizon Trajectories

Why People Wanted It

A final answer can hide how an agent got there. For a long task involving tools, the path matters: which actions it took, what the environment returned, and whether the resulting artifact actually met the task requirements. The real question was not just how to collect more agent text, but how to collect interaction traces whose outcomes can be checked and reused for training.

The Core Move

Sample complete interactions from varied agent setups, run the same task-specific verifier on each, and keep the trace linked to its outcome. The verifier converts an execution into a trajectory-level reward: $r_i^{h,m}=v_i(z( au_i^{h,m}))$, where $z$ is the execution result and $v_i$ is the verifier for task $i$.

A task's acceptance label and a particular trajectory's reward answer different questions. Acceptance says the environment passed construction-time checks; reward says whether this run satisfied the task. A task accepted as skill-dependent does not imply that every agent run succeeds.

Quiz

A task passed construction-time acceptance, but one sampled agent run fails its task-specific verifier. What does the trajectory-level reward represent?

Whether the trajectory's wording resembles a successful demonstration.

Whether this particular execution's outputs and execution state satisfy the task verifier.

A reward shared by every model and harness configuration for that task.

The task's construction-time acceptance label, which guarantees success for the run.

$ check

Why Vary the Harness and Model?

Now the version for readers who want more of the machinery: a sampled agent is induced jointly by its model and harness, so the collected behavior is not attributable to the model alone. Let $P$ be the set of harness–model configurations; sampling a configuration $(h,m)\in P$ in environment $E_i$ produces a trace $\tau_i^{h,m}$.

The harness provides the interaction interface: it determines how actions are issued and environment feedback is received. In the described collection, Codex- and Claude Code-based configurations interact with Docker environments, inspect inputs, execute actions, observe results, and submit an answer or artifact. Varying both harness and model broadens the sources of traces beyond one agent configuration. The trace records the action–observation sequence, not merely the submitted result.

That diversity comes with a tradeoff: traces reflect the combined model–harness setup. What can be compared fairly when the interaction interface changes?

Why the Verifier Scores Outcomes, Not Prose

Here is the part that gets technical: the verifier consumes the resulting outputs and execution state, denoted $z(\tau_i^{h,m})$, rather than judging the wording of the trace. Applying the task-specific verifier gives $r_i^{h,m}=v_i(z(\tau_i^{h,m}))$. In words, the reward is determined by whether this execution's result passes the task's checks.

Keep the two labels separate. The task-level label $\ell_i$ records construction-time acceptance, while $r_i^{h,m}$ is specific to one sampled execution. This distinction prevents a successful environment from being mistaken for a guarantee about every agent. What should training retain once the outcome is known?

Why One Trajectory Record Serves Two Learning Uses

Now the deeper data-model detail: the record binds the task prompt and skill metadata to the environment configuration, actions, observations, final outputs, verifier results, execution logs, and available failure reasons. Keeping these together makes the measured outcome interpretable in light of the procedure that produced it.

Successful traces provide demonstrations for supervised fine-tuning. Failed traces remain useful for analyzing unsuccessful execution and procedural failure modes; they are not successful demonstrations. The accepted environments and executable verifiers also support reinforcement learning with outcome-based rewards. In short, the same interaction record can support imitation when it succeeds and measured reward-based learning from execution outcomes. How much of that structure should be retained rather than collapsing everything to a final score?

Minimal Causal Code

The loop below makes the causal chain visible: run an agent, verify the resulting execution, then store the trace and its outcome. The interfaces are schematic.

records = []

for task in accepted_tasks:
    for harness, model in configurations:
        env = make_environment(task)
        trace = []
        while not env.done:
            observation = env.observe()
            action = run_agent(harness, model, task.prompt, observation)
            result = env.step(action)
            trace.append((observation, action, result))

        output = env.final_output()
        execution_state = env.execution_state()
        reward = task.verifier(output, execution_state)
        records.append({
            "task": task.prompt,
            "skill_metadata": task.skill_metadata,
            "environment": env.configuration,
            "trace": trace,
            "output": output,
            "execution_state": execution_state,
            "verifier_result": reward,
            "logs": env.logs(),
        })

What You Gain, and What You Lose

You gain trace-level supervision grounded in executed work, not just final-answer text; varied model–harness configurations expose more than one interaction pattern; and verifier outcomes can be used as supervised labels or reinforcement-learning rewards. Retaining failures also supports analysis rather than silently discarding every unsuccessful run.

You also inherit the verifier's boundaries: a reward reflects the checks the task defines, not an all-purpose judgment of quality. A trace collected under one harness may encode that harness's interaction conventions, and a task-level acceptance label cannot stand in for run-level success. The approach is therefore a way to make long-horizon experience measurable and reusable—not a guarantee that every trace is correct or every capability transfers.