Scripted Stakeholders and Anti-Gaming Task Validation

Why People Wanted It

A customer-delivery agent is judged in a setting where the customer knows things the task description may omit, and where a plausible-looking answer can still fail the real job. But if the simulated customer behaves differently from run to run, a score can change because the environment changed rather than because the agent improved. And if a task rewards literal compliance, doing exactly what the request says may look successful while missing the constraints that determine the outcome.

So the real question was never just how to score an agent. It was how to make customer interactions reproducible, while admitting tasks only when their scoring distinguishes useful work from doing nothing, blindly following the request, or taking shortcuts.

The Core Move

Fix what stakeholders know and when they disclose it, then validate each task against reference, do-nothing, literal, and shortcut anchors before including it.

For a task score $S$, the ordinary admission checks are $S_{ref} \ge 0.95$, $S_{noop}=0$, $S_{literal} \le 0.2$, and $S_{shortcut} \le 0.8$. In words: a reference solution must score highly, doing nothing must get zero, literal compliance must usually score poorly, and shortcuts must remain bounded.

Why Scripted Stakeholders Make Runs Comparable

Now the version for readers who want more of the machinery: reproducibility is not just using the same customer prompt; it means fixing the information and disclosure behavior that shape what an agent can learn through conversation. FDE-Gym scripts stakeholder responses, maps questions to topics with a small model, and fixes each stakeholder’s knowledge, misconceptions, and conditions for disclosure in advance.

That distinction matters because the same question can produce different evidence if the simulated stakeholder changes. Scripts also encode limited patience, evidence requirements, dependencies on earlier conversations, temporary unavailability, and contact restrictions. The agent must infer roles and communication constraints from customer documents rather than receiving them as a uniform interface. The simulation covers more than 130 stakeholders across 20 scenarios, and changing the user-simulation model can shift measured success rates by several percentage points. What does fixing those rules still leave for task validation to catch?

Why Admission Needs More Than a Good Reference Answer

Here is the part that gets technical: the reference score checks that a task is solvable, while the do-nothing score checks that the grader does not award success without meaningful action. The literal anchor tests whether simply implementing the instruction is enough. Usually it must not be: omitted constraints can still determine whether the result works.

There is a deliberate exception. If the sponsor’s request is both correct and complete, literal compliance is the reference solution, so it must score at least $0.95$ rather than being penalized. This prevents distrust of the sponsor from becoming a strategy that is always rewarded.

The shortcut anchor tests strategies such as overfitting to historical data, guessing round-number thresholds, or sending every uncertain decision to a human; these must score at most $0.8$. Attempts to bypass customer controls or manipulate the grader are capped near zero. The graders were tested against more than 270 shortcut and bypass strategies. If all these anchors pass, what kind of behavior is the task actually selecting for?

Minimal Causal Code

A plain sketch of the admission checks makes their roles visible. The exception for a correct-and-complete sponsor request is explicit; the checks are gates, not a score formula.

def admit_task(reference_score, do_nothing_score, literal_score,
               shortcut_scores, sponsor_request_is_complete):
    if reference_score < 0.95:
        return False
    if do_nothing_score != 0:
        return False

    if sponsor_request_is_complete:
        if literal_score < 0.95:
            return False
    elif literal_score > 0.2:
        return False

    if any(score > 0.8 for score in shortcut_scores):
        return False
    return True

The near-zero cap for control-bypass or grader-manipulation attempts is a separate safeguard, not represented by the shortcut threshold in this sketch.

What You Gain, and What You Lose

Fixed stakeholder scripts make runs more comparable, and the admission anchors make it harder for a task to pass merely because the reference works or because a shortcut exploits the grader. The checks also make an important distinction: literal compliance is usually insufficient, but it is not automatically wrong when the request fully specifies the right solution.

The tradeoff is that scripted disclosure rules are a controlled model of customer interaction, not the full variability of real customers. A stable score can be easier to interpret, but it cannot by itself establish how an agent handles every unscripted stakeholder or delivery setting. Task admission also depends on choosing meaningful reference, literal, and shortcut cases; the anchors test those cases rather than eliminating the need for careful task design.