FDE-Gym: Business-Outcome Scoring

Why People Wanted It

A deployed system can look competent while failing to improve the customer’s actual outcome. And even when it improves a metric, it may do so through costly stakeholder messages or an operational incident. The real question is not simply whether the system works, but whether the delivered change creates worthwhile improvement against a meaningful reference after those costs are counted.

The Core Move

FDE-Gym expresses business improvement in units of an expert reference solution, then subtracts stakeholder-contact costs and applies an incident rule. Its normalized uplift is

$U = \frac{KPI_{delivered} - KPI_{baseline}}{KPI_{reference} - KPI_{baseline}}.$

The baseline is the unchanged customer system, and all three KPI values are evaluated on the same held-out data. A score of zero means no improvement over doing nothing; one means matching the reference solution’s improvement; a negative value means the delivery made the outcome worse than baseline.

Quiz

A delivered system has positive normalized uplift, but an operational incident occurs and the run incurs stakeholder-contact charges. Which describes the single-run score rule?

Use zero as the adjusted uplift, then subtract contact cost; the task is not resolved because an incident occurred.

Keep the positive uplift and subtract contact cost; resolution depends only on whether the score reaches 0.8.

Replace the entire score, including contact cost, with zero whenever an incident occurs.

Floor uplift at negative one and ignore contact cost for incident runs.

$ check

Why the Expert Reference Sets the Scale

Now the version for readers who want more of the machinery: the denominator makes uplift relative to the improvement available in the reference, rather than to the raw size or units of a customer KPI. Let $B$, $D$, and $R$ denote the baseline, delivered, and reference KPI values, with higher values assumed better for this expression. Then $U=(D-B)/(R-B)$: the numerator is the delivery’s gain over doing nothing, while the denominator is the reference’s gain over doing nothing. The same construction applies when the business metric is expressed in domain-specific terms, such as net financial benefit or correctly attributed downtime, provided the metric’s direction is interpreted consistently. If $R=B$, this ratio has a zero denominator and does not define a usable normalization; the scoring setup therefore needs a reference that improves on baseline.

Why Incidents Can Erase Positive Uplift

Here is the part that gets technical: an operational incident changes the uplift before contact costs are deducted. If an incident occurs, positive uplift is set to zero, but negative uplift is retained:

$U_{adj} = \begin{cases} \min(U,0), & \text{if an incident occurs} \\ U, & \text{otherwise.} \end{cases}$

In words, an incident cannot be offset by a business gain, while a system that already harms the baseline keeps that negative result. This is a one-sided adjustment, not a blanket replacement of every incident run with zero.

Why Contacts Count Against the Delivery

Now the cost model adds a second kind of consequence: each chargeable contact contributes its predefined charge, so $C=\sum_{k\in K} c_k$. In words, add the prices for the run’s chargeable stakeholder contacts. The rules can charge more for exceeding a person’s patience or violating contact restrictions; unanswered messages are not charged, and trial runs or days spent waiting can count when those scenario mechanisms apply.

The single-run score is $S=U_{adj}-C$: incident-adjusted uplift minus contact cost. A task is resolved only when $S\geq 0.8$ and no incident occurs. For aggregate reporting, incident-adjusted uplift is floored at $-1$ before the full contact cost is subtracted and task-run scores are averaged; that floor does not alter the single-run score or its resolved-task test. What does this make visible that a raw task-success label would miss? Both the cost of reaching the outcome and the severity of operational failure.

Minimal Causal Code

This loop makes the single-run logic visible. It expects higher KPI values to mean better outcomes and assumes a valid reference denominator.

def fde_run_score(kpi_baseline, kpi_delivered, kpi_reference,
                  contact_charges, incident):
    uplift = (kpi_delivered - kpi_baseline) / (kpi_reference - kpi_baseline)

    # An incident removes gains, but preserves negative outcomes.
    adjusted_uplift = min(uplift, 0.0) if incident else uplift
    contact_cost = sum(contact_charges)
    score = adjusted_uplift - contact_cost
    resolved = score >= 0.8 and not incident

    return score, resolved

What You Gain, and What You Lose

The score makes a delivered system answer to more than a proxy notion of task completion: it measures improvement against the unchanged customer system, calibrates that improvement to an expert reference, and charges for interactions and incidents. The resolved criterion is explicit, and the separate aggregate floor prevents very large negative adjusted uplift from dominating the reported average while leaving full contact costs in place.

The tradeoff is that the result depends on the chosen business KPI, the quality and applicability of the expert reference, and the task’s predefined contact and incident rules. Normalization does not make unlike business outcomes interchangeable, and a scalar score can hide which component drove a result. FDE-Gym therefore gives a structured accounting rule, not a substitute for examining the underlying KPI and run-level costs.