First-Signal Labels for Diagnosing Agent Failures
Agent failures can look alike in a final score; labeling each trajectory by its first observed failure category separates unsupported plan claims, harmful harness interventions, process breakdowns, and poor delivered plans.
Read the paper ↗What it is
- First-signal label A trajectory is classified by the first observed kind of failure, rather than being counted only as a generic bad run.
- Four failure categories F1 is an unsupported plan claim; F2 is a harmful runtime intervention; F3 is looping or running out of steps; F4 is a poor plan that was delivered.
Why it matters
Separating these signals helps distinguish plan-content problems from execution-process problems. It makes failure composition visible instead of hiding different breakdowns inside one score.
Before getting into the labels, here is the short version: the method tags a failed agent run by the first kind of failure you can observe. That turns a generic “bad answer” into a clue about whether the trouble was in the plan, in the runtime process, or in the plan that eventually got delivered.
The canonical example is a travel-planning run: the benchmark scores a proposed itinerary, while the trajectory shows how the agent got there. The score tells you the result; the first-signal label tells you what went wrong along the way. The categories generalize beyond that example as a way to describe trajectory failures, but the excerpt’s concrete setting is travel planning.
-
Unsupported claims: sometimes the plan asserts something the trajectory does not back up. That is F1. Think of a notebook where an entry is written down without a source; the analogy is limited because the label concerns a claim in the plan, not the reliability of every item in a notebook.
-
Runtime and execution trouble: the harness can intervene in a way that gets in the agent’s way (F2), while a run can also loop or exhaust its step budget (F3). Those may both stop a useful result, but they are not the same diagnosis: one names a harmful harness intervention, the other names a trajectory that gets stuck or runs out of steps. What do we call a run that completes but still gives a bad plan?
-
Poor delivered plans: F4 is for a plan that makes it through delivery but is simply poor. Delivery and quality are separate: reaching the end does not certify the result. In the paper’s overview, F2 and F3 are process failures, while F4 is a content failure. Does that mean the label itself tells you exactly what to change? No; it identifies the observed category, not a guaranteed remedy.
The rule is to label each trajectory by its first failure signal. Put simply, later trouble does not replace the earlier diagnosis. The excerpt names the four categories but gives no formula for the labeling rule. That matters because a final score alone cannot distinguish a failed process from a poor completed plan.
Read the deep dive
Why People Wanted It
A low score tells you that an agent failed, but not what kind of failure came first. A plan may contain claims the trajectory does not support; a runtime intervention may get in the way; execution may loop or run out of steps; or the agent may successfully deliver a plan that is simply poor. Treating all four as one bucket makes it harder to tell whether the problem is in the plan, the runtime process, or the delivered result.
So the real question is not just “Did the run fail?” It is “What was the first observed failure signal?” The taxonomy gives each trajectory one first-signal label, F1–F4, so those distinctions can be measured separately.
The Core Move
Assign a failed trajectory to the first failure signal observed: F1 for unsupported assertions in the plan, F2 for harmful harness interventions, F3 for loops or exhausted step budgets, and F4 for a delivered but poor plan.
The source gives this taxonomy in words; it does not provide a formula for assigning the labels.
Why the First Signal Separates Plan Claims from Runtime Harm
Now the version for readers who want more of the machinery: F1 and F2 distinguish a problem in what the plan asserts from a problem caused by the harness acting during execution. F1 is an unbacked assertion in the plan; F2 is an intervention by the harness that gets in the way. The label is about the observed failure signal, not a general verdict that the model or runtime is bad. The boundary matters because changing plan generation and changing runtime behavior are different diagnostic moves.
Why Loops and Exhausted Budgets Are Their Own Class
Here is the part that gets technical: F3 marks a process breakdown—loops or exhausted step budgets—rather than an unsupported claim or a poor completed plan. The taxonomy treats those as the same category, but keeps them distinct from F2: a harmful intervention is attributed to the harness, while looping or using up the step budget is the observed trajectory failure. This classification says what signal appeared, not why the run entered that state. What should the label say when execution finishes but the result is still bad?
Why a Delivered Poor Plan Is Different
Now we can make the final distinction precise: F4 is a delivered plan that is simply poor. Delivery is not the same as quality. A trajectory can get to an answer and still belong to the content-failure class; conversely, a run that never gets through execution is not an F4 merely because its final score is low. In the paper’s overview, F2 and F3 are process classes, while F4 is a content class. Does separating these labels guarantee the right intervention? No: the labels diagnose observed failure signals; they do not themselves fix the cause.
Minimal Causal Code
The paper excerpt defines the categories but does not specify an implementation algorithm. This deliberately plain loop makes the first-match rule visible; the category predicates must come from the trajectory annotation protocol.
# signals are ordered observations from one trajectory.
# Each item has a category in {"F1", "F2", "F3", "F4"} or None.
def first_failure_label(signals):
for signal in signals:
if signal.category in {"F1", "F2", "F3", "F4"}:
return signal.category
return None
What You Gain, and What You Lose
You gain a decomposition that distinguishes unsupported plan assertions (F1), harmful harness interventions (F2), loops or exhausted step budgets (F3), and delivered poor plans (F4). In the paper’s overview, F2 and F3 are process failures and F4 is a content failure. This makes failure composition a more informative diagnostic than a single aggregate score.
You also inherit the limits of first-signal labeling. Later problems in a trajectory are not represented by its first label, and the categories describe observed signals rather than proving their underlying cause. The taxonomy is a measurement choice, not a guarantee that any particular remedy will work. The paper’s separate harness-evolution loop uses these labels to diagnose trajectories; this module covers the labels, not that loop.
A final failure score says the run went wrong; its first signal says what kind of wrong appeared first.