Judge-Verified Harness Evolution
Improve a frozen model’s prompts, tools, and runtime through measured edits—and distinguish a noisy score bump from a real gain or a reason to train the weights.
Read the paper ↗What it is
- Harness evolution A language model proposes edits to the prompts, tools, and runtime around a model whose weights stay fixed.
- Judge verification Benchmark evaluation checks proposed edits before acceptance. Repeated runs and a refreshed reference help keep random score bumps from steering the search.
- The limit of the loop The search depends on the feedback it receives. It can improve how the system works while leaving poor answers unresolved; training the model’s weights is a separate lever.
Why it matters
You can improve a model-powered system without retraining it, but a better setup is not the same as a better-trained model. Separate selection tasks from unseen evaluation tasks, and inspect what failures remain before deciding what to change next.
The short version is that you can improve the setup around a model without changing the model itself. A language model suggests edits, and benchmark evaluation checks whether they help. The catch is that noisy scores and incomplete feedback can make this search look more successful—or more complete—than it really is.
-
Change the setup first: The harness means the prompts, tools, and runtime surrounding the model. On DeepPlanning, a travel-planning benchmark, that might include a state ledger that keeps planning information available. This changes what the model can work with, not its learned parameters. Think of giving an engineer a better notebook rather than teaching them a new skill; the analogy stops at the fact that the model must still interpret and use the notebook. So how do we tell a genuinely useful edit from a convincing suggestion?
-
Measure before accepting: The proposer generates a candidate harness; the judge checks it. Here, benchmark scoring is deterministic and programmatic, not simply another model saying that an edit looks good. But deterministic scoring does not make the model’s behavior deterministic: different runs can produce different travel plans and therefore different scores. Each configuration is measured using four runs, differences below 0.035 are treated as noise, and an accepted change is followed by a fresh anchor—a new reference measurement. I find that distinction essential: testing the output reliably is not the same as producing the same output reliably. What protects the loop from merely fitting the tasks used to choose edits?
-
Keep selection separate from validation: The loop makes decisions on 48 development tasks. Another 72 held-out tasks remain unseen by the loop and test whether the selected setup helps elsewhere. This is the same separation you want between tuning a configuration and testing it: a development winner still has to earn its held-out result. But even a held-out gain leaves another question: what did the harness actually fix?
Why a better harness can still hit a ceiling
A travel planner can fail to finish, or it can finish and deliver a poor plan. Those are different problems. Harness evolution reduced process failures such as loops, but poor delivered plans—the category called F4—remained.
The feedback explains an important limitation. The package shown to the proposer emphasizes measurements of enforcement mechanisms, such as gates and checkpoints, but does not aggregate the scorer’s failure composition. In other words, it does not summarize which kinds of failure dominate. Meanwhile, component-removal tests indicate that capability-granting additions, including the ledger, a distilled lessons file, and invented tools, carry the gains; enforcement components are mostly within noise.
I would not read that ceiling as proof that every possible harness has been exhausted. It is a limit of this search with this feedback. Training weights on trajectories—records of task execution—from the evolved harness is a separate intervention. Testing trained and untrained models under both the original and evolved setups then asks the useful final question: which improvements stay with the model, and which still require the runtime?
Read the deep dive
Why People Wanted It
A model-powered travel planner can fail because it gets stuck in a loop, because its runtime gets in the way, or because it delivers a poor plan. These failures do not necessarily call for the same intervention. Before changing the model’s weights, you can try changing its harness: the prompts, tools, and runtime that surround it.
The problem is the price of finding out whether an edit helps. Long task executions cost evaluation effort, and their scores vary between runs. A plausible proposal and a higher score on one run are not enough.
So the real question was never just whether a model could rewrite its setup. It was whether a propose-and-verify loop could find gains that survive repeated measurement and held-out evaluation—and what failures would remain afterward.
The Core Move
Keep the model weights fixed, let an LLM propose a harness edit, and use benchmark evaluation rather than the proposer’s confidence to decide whether to keep it.
Call the current harness and a proposed replacement . A rollout is a task execution under a configuration. Let be the benchmark score for rollout repetition . The reported protocol measures each configuration with a four-rollout mean:
In words, compare average measured performance, not descriptions of the edits. Differences below 0.035 are treated as noise, and every accepted change is followed by a fresh reference measurement, called an anchor.
“Judge” names the verification role here; it need not mean an LLM giving an opinion. DeepPlanning uses deterministic programmatic scoring of the generated plans.
Why Deterministic Scoring Still Needs Repeated Runs
Here is the part that gets technical: a deterministic scorer removes variation in how a fixed output is graded, not variation in which output the model produces.
An unchanged harness can generate different trajectories and plans across runs. The reported per-rollout standard deviation is 0.023, which motivates treating small observed differences cautiously. The four-rollout mean and the 0.035 noise threshold belong to this experimental protocol; they are not universal constants for harness search.
A fresh anchor after acceptance remeasures the reference instead of carrying the accepted candidate’s selection-time score forward indefinitely. That matters because selecting a candidate for a favorable measurement can also select favorable noise. Refreshing the anchor does not eliminate uncertainty, but it avoids treating that selected measurement as an unquestionable baseline.
The causal state transition is therefore conditional: rejection leaves the harness unchanged; acceptance replaces it and refreshes the measurement used as a reference. Even then, how do you know the edit helps beyond the tasks that selected it?
Why Selection and Held-Out Measurement Must Stay Separate
Now the version for readers who want more of the machinery: repeated evaluation addresses rollout variation, while a held-out split addresses a different problem—choosing edits that fit the selection tasks.
Every decision uses 48 development tasks. The 72 held-out tasks are never exposed to the loop. Candidate generation and acceptance can therefore adapt to development feedback, but not to held-out feedback.
This separation is necessary even with a deterministic scorer. Repeatedly choosing edits against the same tasks makes those tasks part of the optimization process. A held-out evaluation asks whether the resulting harness helps on tasks outside that process.
For example, Qwen3.5-4B gains 0.143 on development tasks and 0.133 on held-out tasks relative to the seed harness. These are results for that model and setup, not an expected return from any proposed edit. The seed harness is the starting configuration; the evolved harness is the configuration produced by the search.
A held-out gain establishes a better measured system, but it does not yet identify whether the improvement lives in the runtime or in the model. What failure types did the search leave unresolved?
Why the Feedback Can Leave Poor Plans Unresolved
One level deeper: the proposer can only target failure mechanisms that its feedback makes legible, so judge verification and useful diagnosis are not the same thing.
A delivered but poor plan is labeled F4. This differs from process failures such as loops or exhausted step budgets. Harness evolution reduces process failures, but F4 remains: its share rises from 10% to 28% of Qwen3.5-4B trajectories and stays at 25% for Qwen3.5-9B. These are shares of trajectories assigned to a failure category, not measurements of how bad each individual plan is.
The proposal model’s evidence package leads with metrics for gates and checkpoints, which enforce aspects of the workflow. It does not aggregate the scorer’s failure composition into a summary of which failure types dominate. A performance score can therefore verify progress without telling the proposer what is causing the remaining poor plans.
Component ablations—tests that remove individual components—also distinguish enforcement from added capability. Exit gates, checkpoints, and hard blocks are mostly within noise; a state ledger, a distilled lessons file, and invented tools carry the gains in the reported comparisons. More enforcement is not automatically the useful next edit.
Weight training is a separate intervention: trajectories collected under the evolved harness can be used to update model parameters. Crossing seed and evolved harnesses with base and trained weights separates gains retained under the original runtime from gains that still depend on the evolved runtime. The remaining question is not whether all harness possibilities are exhausted, but whether the unresolved failure calls for better feedback, another runtime capability, or weight training.
Minimal Causal Code
This plain PyTorch loop makes the harness replacement and anchor refresh visible. It illustrates a conservative above-noise acceptance rule, not a complete statistical test. The evaluator runs development tasks only; held-out evaluation happens after selection.
import torch
def mean_score(harness, tasks):
# Each call runs and programmatically scores the task set.
scores = []
for _ in range(4):
scores.append(run_and_score(harness, tasks))
return torch.tensor(scores, dtype=torch.float64).mean().item()
harness = seed_harness
anchor = mean_score(harness, dev_tasks)
noise_threshold = 0.035
# The underlying model weights stay fixed throughout this loop.
for _ in range(proposal_budget):
feedback = build_proposal_feedback(harness, dev_tasks)
candidate = propose_edit(harness, feedback)
candidate_score = mean_score(candidate, dev_tasks)
improvement = candidate_score - anchor
if improvement > noise_threshold:
harness = candidate # Update the system state.
anchor = mean_score(harness, dev_tasks) # Fresh reference.
# Otherwise keep the current harness and reference.
# Do not feed this result back into candidate selection.
held_out_score = mean_score(harness, held_out_tasks)
The benchmark execution, feedback construction, and proposal generation are abstract functions. The important distinction is explicit: the candidate’s selection score is not reused as its fresh anchor.
What You Gain, and What You Lose
You gain a way to improve the surrounding system without updating weights. Evaluation filters proposals, repeated runs reduce reliance on individual outcomes, and held-out tasks test whether selected changes transfer beyond development decisions.
You pay for repeated evaluation, and you remain limited by the edit space and feedback available to the proposer. A judge can measure a score without supplying a diagnosis. The observed ceiling therefore belongs to this search and its feedback, not to every conceivable harness.
Finally, an improved harnessed system is not a trained model. Testing base and trained weights under both seed and evolved harnesses is what distinguishes improvements retained by the model from improvements that still require the changed runtime.
Treat a harness edit as a testable hypothesis: verify it against rollout noise, validate it on unseen tasks, and distinguish a better runtime from learned model improvements.