FDE-Gym: Constructing Customer-Delivery Tasks

Why People Wanted It

A delivery agent can look capable when the task statement hands it the problem, the rules, and the success criteria. Real customer work often does not. A sponsor may describe a symptom, omit a constraint, or ask for a change that would not solve the underlying problem.

So the real question is not only whether an agent can implement a solution. Can you tell whether it can do that when the request is incomplete or pointed at the wrong issue? A useful benchmark needs to separate missing information from missing engineering skill.

The Core Move

Build tasks from expert-adapted customer projects, then vary the instruction and key facts while keeping the operational setting fixed. The central control is simple: change what the agent is told, not the customer environment, stakeholders, or grader.

The benchmark uses four instruction levels, from a fully specified problem at L0 to the sponsor’s original request at L3. The source describes this construction and its levels; it gives no construction formula, so none is needed here.

Why Fixed Environments Make Instruction Levels Informative

Now the version for readers who want more of the machinery: the comparison is interpretable because the workspace, stakeholders, and grading criteria stay the same across all four levels, while only the task instruction changes. If performance shifts, the intended variable is information available in the instruction—not a different customer or a different scoring rule.

At L0, the instruction states the underlying problem, applicable rules and constraints, and how the result is scored. The agent still has to design and implement an effective solution. At L1, it gets a clear problem statement but must determine the rules and constraints. At L2, the request is vague but points in the right direction, so the agent must clarify the problem as well as its constraints. At L3, it receives the sponsor’s original request, which may target the wrong problem; it must judge whether that request addresses the underlying need.

This makes the levels a progression of discovery demands, not a progression of easier and harder codebases. But what if the facts themselves—not just the wording—change?

Why Scenario Variants Change the Answer

Here is the part that gets technical: each scenario is built from a real-world delivery project adapted by industry experts, and it yields an original task plus variants. Tasks in a scenario share a customer environment but can differ in key facts: which rule is outdated, who has critical information, or whether the sponsor’s request addresses the actual problem.

That distinction matters. Rephrasing the same task tests robustness to wording; changing an underlying fact can change which solution is correct. A solution that works for one variant may fail for another, so repeating a familiar pattern is not enough—the agent has to inspect the particular task’s evidence. The benchmark contains 20 scenarios across 18 sectors and 95 tasks; those are properties of this benchmark, not a recipe or equation. How do you keep such a varied set grounded in real delivery rather than invented puzzles?

Why the Tasks Start from Delivery Projects

Now for the construction layer: experts adapted real customer delivery projects carried out in China, Singapore, and the United States into scenarios. The tasks call for an agent, workflow, or service that operates in the customer’s environment. Relevant regulations, standards, and public sources are documented; company and individual identities are de-identified or fictional, and some data are adapted from industry sources.

The point is not to reproduce a customer engagement wholesale. It is to preserve the practical ingredients that make delivery difficult—an operational system, customer context, and facts that matter to the solution—while making tasks suitable for evaluation. Other parts of FDE-Gym address scoring and stakeholder simulation; they are not the construction move explained here.

Minimal Causal Code

There is no executable benchmark-construction algorithm specified in the source passage, so a fictional implementation would suggest more than it establishes. The causal structure can instead be shown as a plain data transformation:

# Illustrative structure, not benchmark source code.
scenario = adapt_delivery_project(project)

for variant in scenario.variants:
    task = make_task(
        environment=scenario.environment,
        stakeholders=scenario.stakeholders,
        grader=scenario.grader,
        underlying_facts=variant.facts,
    )
    for level in ("L0", "L1", "L2", "L3"):
        task.instruction = render_instruction(variant, level)
        save(task)

The visible control is what stays fixed—the operational setting and grader—and what varies: facts across scenario variants and information across instruction levels.

What You Gain, and What You Lose

This design lets you ask whether an agent can implement a known solution, discover constraints, clarify a vague problem, or challenge a misdirected request without changing the surrounding environment between levels. Variants also make it possible for two tasks in one scenario to require different delivery solutions.

The tradeoff is that construction depends on expert adaptation and careful selection of facts; a benchmark task is a designed representation of a delivery project, not the customer engagement itself. Shared settings make instruction-level comparisons cleaner, but they do not make every real-world ambiguity disappear. And changing facts tests whether an agent attends to task-specific evidence, not whether it can handle every kind of customer work.

Quiz

What changes across FDE-Gym’s four instruction levels?

The production system and the evaluation criteria.

The stakeholder set and the customer environment.

The task instruction, from a fully specified problem to the sponsor’s original request.

$ check

Quiz

What distinguishes a scenario variant from merely changing an instruction level?

A variant is simply another name for the L0-to-L3 instruction progression.

A variant can change key facts and therefore may require a different solution.

A variant always changes the grader while keeping all facts fixed.

$ check