Confidence-Thresholded Delegation

Harness choices can save an LLM call but a wrong choice can derail later steps; JevAgentHarness turns choices into typed questions and delegates only when a per-choice confidence threshold fitted on agent trajectories is met.

Read the paper ↗

The short version

What it is

Typed choice The harness turns a choice it already knows how to make—such as selecting a tool or a listed argument—into a question with explicit options.
Confidence-gated handoff A decision model handles the choice only when its confidence clears a cutoff set for that kind of choice; otherwise the large language model decides.

Why it matters

This keeps the fast model away from choices where its uncertainty makes delegation a poor bet, while still avoiding an LLM call on choices that meet the fitted confidence rule.

Before we get into the moving parts, here is the idea in plain terms: let a small decision model handle the agent’s routine, listed choices when it seems sure, and send uncertain choices to the LLM. The point is not to replace the LLM everywhere; it is to decide which handoffs are worth making inside the harness.

A choice has consequences: The canonical example is choosing a tool. If the agent calls the wrong tool, it changes the state the next step sees; this is not like getting one quiz answer wrong and then moving on. So accuracy on isolated questions is not enough to tell us whether delegation is safe.
The harness already knows the options: For a tool choice, the harness has a list of tools available at that step. It can ask the decision model to choose among those options instead of asking it to write a whole response. The same pattern works for listed argument values or a yes/no choice. The limit is important: a free-text argument is not naturally a choice from an enumerated list, so this path may not cover it.
Confidence is a routing signal, not a promise: The decision model returns probabilities for the options and selects the likeliest one. In the cascade, the harness lets that answer through only if the model’s confidence clears a threshold for that kind of choice; otherwise the LLM decides. Think of it as a delegated task with a confidence gate, not a blanket promotion to authority. The notebook analogy is useful here: you can hand off a page you trust, but unlike a notebook, a bad choice can alter the world the agent encounters next.

So why not use one confidence cutoff for everything? A tool choice, an argument choice, and a loop-control choice can have different error costs and different savings. The system therefore fits thresholds per decision point using separate agent trajectories where the decision model answers silently. This makes it possible to compare its confidence and accuracy for each kind of choice and weigh a saved LLM call against the cost of a mistake. The threshold is a practical routing boundary; confidence need not be a perfectly calibrated probability.

For example, the reported fitted thresholds for tools and arguments with JEV and Flash were 1.00 and 0.99, respectively—so in that setup, only near-certain answers were accepted. Those values are specific to the tested setup, not a universal setting. I like the shape of the rule because it makes the fallback explicit, but threshold fitting can only reflect the trajectories and costs it measures.

The paper also describes other ways to combine the models: one can always replace the LLM, shortlist choices for it, guard its proposal, or decide whether an optional step runs. The idea here is narrower: the CASCADE mode, where confidence decides who gets the last word. If a delegation works alone, will it still help when the agent makes several decisions in sequence?

Read the deep dive

Why People Wanted It

A harness repeatedly makes choices the LLM could make: which tool to call, what argument to pass, or whether to take an optional step. A fast decision model can answer these as listed options, but a wrong answer is not an isolated classification error: the agent may act on it, changing what it sees next. Always routing choices through the LLM is safer in some cases, but pays for an LLM call every time.

So the real question was never just whether the decision model is accurate. It was: which kinds of choices can this harness safely hand off, and when should it keep the LLM in control?

The Core Move

Turn each harness choice into a typed multiple-choice question, then delegate it only when its confidence clears a threshold fitted separately for that kind of choice on the agent’s own trajectories.

For a decision point $d$ in state $s$ , the cascade delegates when $c(s) \geq \tau_d$ ; otherwise the LLM decides. Here $c(s)$ is the decision model’s confidence in its selected option, and $\tau_d$ is the fitted threshold for that decision point.

Quiz

In the confidence-thresholded cascade, what happens when a decision model’s confidence is below the threshold for that decision point?

The harness executes the decision model’s top option anyway.

The decision model generates a free-form explanation that replaces the choice.

The LLM makes the choice as usual.

The agent stops the run.

Why Harness Choices Need Types

Now the version for readers who want more of the machinery: the interface is not a generic text prompt but a structured choice whose available options come from the harness. A decision point is a recurring kind of choice, such as tool selection or a fixed-list argument; at a particular state, the harness supplies the options available there. The question carries the task, history, and options, and the decision model returns a probability for each option rather than composing free-form text.

That structure matters because a tool choice and a copied text span are not interchangeable output types. The harness can validate and execute the chosen option in the form it expects. In the reported setup, tool calls are decomposed into decisions such as selecting a tool and supplying its arguments; fixed-list or yes/no values and spans copied from context can also be represented. Free-text arguments do not fit this listed-choice path and may require the LLM. The practical boundary is therefore set by what the harness can enumerate as options.

Why Thresholds Belong to Each Decision Point

Here is the part that gets technical: a single global confidence cutoff ignores that different choices have different error costs and different savings. The fitted rule uses trajectories in which the decision model answers silently while the agent runs. Those trajectories provide per-decision-point observations used to estimate the LLM’s correctness, the price of a saved call, the impact of a wrong choice on task success, and how the decision model performs at different confidence levels.

The paper gives a decision-theoretic threshold when confidence tracks correctness: $c(s) \geq q_L - \Delta C/V$ . In words, delegate when confidence is at least the LLM’s chance of being right, minus the saved call’s value relative to the cost of a wrong choice. In fitting, the decision model’s empirical accuracy at each confidence is also used, so the selected threshold need not equal a literal calibrated probability boundary. If no confidence region makes delegation worthwhile, the LLM retains that decision point.

Why Delegation Is a Cascade, Not a Replacement

Now the deeper operational distinction: the threshold governs the CASCADE mode, not every possible way a decision model can interact with a harness. In this mode, the decision model answers first; confidence above the decision point’s threshold lets its option proceed, while lower confidence sends that choice to the LLM. The fallback is local to the choice, so the harness does not need to replace the LLM across the whole run.

Other integration modes exist, including always replacing the LLM, shortlisting options for it, guarding its proposed answer, or triggering an optional step. They are not the confidence-thresholded cascade described here. The key deployment question is: once a delegation passes on its own, does it remain useful alongside the other choices the agent makes?

Minimal Causal Code

This plain loop makes the handoff visible. The threshold lookup is per decision point; the harness still owns the option list and execution.

def choose(harness, decision_model, llm, state, decision_point, options, thresholds):
    question = harness.make_typed_question(state, decision_point, options)
    option, confidence = decision_model.answer(question)

    if confidence >= thresholds[decision_point]:
        return option  # harness executes the delegated choice

    return llm.choose(state, decision_point, options)

What You Gain, and What You Lose

You can avoid an LLM call on choices where the decision model is confident enough under a threshold fitted for that choice, while keeping the LLM as the fallback. The typed interface makes the handoff concrete: listed tools, argument values, or context spans can be returned as choices the harness understands.

The tradeoff is that confidence does not erase the cost of a wrong action. A choice can change later states, so a threshold fitted from trajectories is a deployment rule, not a guarantee for every new task. Fitting also depends on running the agent in shadow mode on separate tasks and measuring the relevant costs and outcomes. The method does not make all decisions safe to delegate: if no threshold qualifies, the LLM keeps deciding. The broader system also trains decision models on states agents visit; that is a separate module’s concern, not the delegation rule itself.

Quiz

Why fit a separate confidence threshold for each decision point?

Because the LLM is never used after fitting.

Because the value and risk of delegating can differ by kind of choice.

Because every decision point must use a different model.

Because thresholds make the decision model’s options unnecessary.

Core Takeaway

Delegate a choice only when its own fitted confidence bar is cleared; otherwise let the LLM keep the last word.