- The problem
- Agents that use large language models for long tasks can be slow and costly because the large model makes nearly every small choice. A fast, smaller model may make mistakes, and one bad choice can change what the agent faces next.
- The idea
- The paper develops a way to test fast choice-making models both on sample questions and while they operate an agent, where their actions affect later steps. Its harness lets the fast model handle a choice only when its confidence passes a threshold learned from that agent’s own runs; otherwise, it asks the large model. The authors also train a compact model on the situations agents actually encounter, including situations following an error.
- What it got
- On unfamiliar tasks, agents using the trained compact model solved 91.5% of tasks, compared with 82.0% using JEV. On familiar task types, trained compact models cut costs by 81.1–89.0% and made agents up to 2.5 times faster without losing 3 or more percentage points of task success.