Harness Evolution Hits a Ceiling: When Weight Training Should Begin

Yuan Tian, Bing Hu, Hao Wang, Binghang Lu, Fang Wu

arXiv 2610.11655 21 pages Read the paper

The short version

The problem: Long-running language-model agents can get stuck, fail to finish, or finish with a poor answer. A developer may need to change the surrounding software or train the model, but a score alone does not show which fix fits the problem.
The idea: The paper labels each failed run by the first sign of trouble, distinguishing process problems—such as repeated actions or an unfinished task—from a poor answer that was delivered. It uses those labels to study two fixes: letting a language model propose software changes that a judge checks, and training the model on runs made with the improved software. It then tests whether the training still helps with the original software in place.
What it got: On unseen planning tasks, the software-change process raised the smaller model’s score by 0.133. Training on runs from the improved software added about 0.13 for both tested model sizes when evaluated with the original software.

Modules from this paper

First-Signal Labels for Diagnosing Agent Failures

Agent failures can look alike in a final score; labeling each trajectory by its first observed failure category separates unsupported plan claims, harmful harness interventions, process breakdowns, and poor delivered plans.

agentsevaluationfailure-analysisharnesses

Judge-Verified Harness Evolution

Improve a frozen model’s prompts, tools, and runtime through measured edits—and distinguish a noisy score bump from a real gain or a reason to train the weights.

agentsharnessesevaluationself-evolution