Family-Calibrated Rewards

Why People Wanted It

A terminal agent can pass scenario checks and still fail the final completion gate. Reward only the gate, and you discard information about partial progress. Reward passed checks without distinguishing completion, and a check-rich failure can look better than a successful run.

The problem is not just sparse feedback. It is the meaning of the feedback. So the real question is how to preserve useful progress credit while making verified completion a separate, decisive signal.

The Core Move

Scale passed checks by task family; for success prioritization, bound the progress term and add a strict-success bonus without discarding that term.

Let $P\geq0$ count passed non-hygiene checks, $S\in\{0,1\}$ indicate strict terminal success, and $f$ identify the task family. Let $V$ mean that valid Common Test Report Format (CTRF) grading is available. With positive family divisors $D_f$ and $D'_f$, the rules are:

$$R_{FDR}=\begin{cases} \max(0,P/D_f), & V,\\ S, & \text{otherwise}, \end{cases} \qquad R_{SPR}=\begin{cases} S+P/(P+D'_f), & V,\\ S, & \text{otherwise}. \end{cases}$$

Since $P\geq0$, FDR's lower clamp is redundant. With valid grading, FDR scores progress regardless of success; SPR scores progress plus success. Neither rule switches to a flat success reward when valid grading is present.

Grading and outcome	FDR	SPR
Valid report, failure	$P/D_f$	$P/(P+D'_f)$
Valid report, success	$P/D_f$	$1+P/(P+D'_f)$
No valid report	$S$	$S$

Quiz

With valid grading, how is a strictly successful rollout scored?

Both return only the raw number of passed checks.

FDR adds one to progress; SPR discards progress and returns one.

Both rewards discard progress credit and return exactly one.

FDR keeps its family-scaled progress score; SPR adds one to its bounded progress score.

$ check

Why Family Calibration Changes What a Check Is Worth

Now the version for readers who want more of the machinery: the family divisor makes each passed check worth a fixed fraction of a historical check-count scale, not a fraction of the current rollout's executed checks.

For FDR, define

$D_f=q_{0.90}(N\mid f)>0,$

where $N$ is the total number of non-hygiene calibration checks and the quantile is computed within family $f$. The fallback divisor is $82$. The lookup is frozen within a training phase.

For valid grading, adding a passed check changes FDR by

$R_{FDR}(P+1)-R_{FDR}(P)=\frac{1}{D_f}.$

Changing only $S$ does not change FDR at all. This is what “no separate completion bonus” means: successful rollouts retain their check-based score rather than being assigned $1$.

Nor does the historical divisor cap the reward. If $P>D_f$, then FDR exceeds $1$, whether the rollout succeeds or fails. There is no uniform upper bound over check counts. How can a reward keep progress information without allowing a failure to overtake a success?

Why Success-Prioritized Reward Keeps a Separate Success Tier

Here is the part that gets technical: SPR adds the success indicator to a bounded progress function on both successful and failed rollouts.

For a positive recalibrated divisor $D'_f$, define

$g_f(P)=\frac{P}{P+D'_f},\qquad R_{SPR}=S+g_f(P)$

when grading is valid. The progress function is increasing and concave:

$g'_f(P)=\frac{D'_f}{(P+D'_f)^2}>0,\qquad g''_f(P)=\frac{-2D'_f}{(P+D'_f)^3}<0.$

Each additional check helps, but its marginal contribution shrinks. Since $0\leq g_f(P)<1$, the outcome tiers are

$$S=0\implies 0\leq R_{SPR}<1,\qquad S=1\implies 1\leq R_{SPR}<2.$$

The fallback $R_{SPR}=S$ preserves these tiers when grading is unavailable. Holding check count, family, and report validity fixed, success adds exactly one unit. Across different rollouts, however, there is no uniform positive separation: a failure's progress score can approach $1$, while a success with no passed checks scores exactly $1$.

This addresses an observed proxy mismatch: among $4{,}352$ audited trajectories, failures received $50.09\%$ of FDR reward mass, $158$ failures scored at least one, and $48$ trajectories passed every executed scenario check but failed the terminal gate.

The bound $0\leq R_{SPR}<2$ implies reward variance at most one. It does not establish lower policy-gradient variance, convergence, equivalence to maximizing success probability, or potential-based policy invariance. What determines how strongly progress is damped within each family?

Why Frozen Recalibration Targets Families, Not Rollouts

Now the deeper calibration detail: SPR changes the family lookup offline rather than letting each rollout or minibatch determine its own normalization.

Recalibration doubles divisors for high-reward-failure families and quarters them for long, low-reward families. Failure suppression takes precedence. Increasing $D'_f$ reduces the progress term for a fixed $P$; decreasing it increases that term. Neither change alters the separate success bonus.

The resulting lookup is frozen within the phase. This is neither per-batch normalization nor a bonus based on the current trajectory's length. It adjusts how much check progress is worth for a family while leaving the success/failure tier boundary intact. The remaining question is whether a fixed lookup remains appropriate as the policy's failure modes change.

Minimal Causal Code

import torch


def terminal_reward(checks, strict_success, valid_ctrf, divisor, mode):
    # checks contains (passed, is_hygiene) pairs from a completed report.
    # divisor is selected from the appropriate frozen family lookup.
    if mode not in ("fdr", "spr"):
        raise ValueError(mode)

    success = torch.tensor(float(strict_success), dtype=torch.float64)
    if not valid_ctrf:
        return success

    d = torch.tensor(float(divisor), dtype=torch.float64)
    if d.item() <= 0:
        raise ValueError("family divisor must be positive")

    # Make the progress-count state update explicit.
    passed_count = torch.zeros((), dtype=torch.float64)
    for passed, is_hygiene in checks:
        increment = float(bool(passed) and not bool(is_hygiene))
        passed_count = passed_count + increment

    if mode == "fdr":
        # Success does not replace or add to valid-report FDR.
        return torch.clamp_min(passed_count / d, 0.0)

    progress = passed_count / (passed_count + d)
    # Successful rollouts KEEP progress credit and ADD one.
    return success + progress

Use $D_f$ for FDR and $D'_f$ for SPR. The loop exposes count accumulation; it does not emit intermediate rewards. Both rewards are computed once at rollout completion, and terminal feedback must not be duplicated across computational chunks.

What You Gain, and What You Lose

FDR gives simple, constant marginal credit on a historical family scale. It does not explicitly reward completion, and a failed rollout can earn a high score. Successful rollouts are scored by the same progress rule.

SPR keeps progress credit on every validly graded rollout and adds a separate success bonus. This ensures every failure scores below every success, but introduces diminishing marginal credit and no uniform positive margin between the tiers. Reward ordering alone does not establish success-probability maximization or optimization stability.

Frozen recalibration can reduce partial credit in families associated with misleading failure rewards, but it relies on an offline lookup. If valid grading is unavailable, both methods lose partial-progress resolution and return only strict success. In either case, “dense” describes the resolution of a terminal score—not step-level reasoning supervision.

Quiz

Why does SPR's reward bound not by itself guarantee lower policy-gradient variance or convergence?

SPR is computed at every generated token, so its variance cannot be measured.

Strict-success rewards are unbounded.

Reward variance bounds do not establish gradient-variance ordering or convergence.

The family divisor changes on every rollout, making all comparisons invalid.

$ check