GracefulRL: Learning to Fall Back Without Getting Stuck

Why People Wanted It

In ordinary verifier-based reinforcement learning, a trajectory is either a success or an error. But many tasks have a third sensible outcome: stop, abstain, restore a previous state, or hand the task to another agent. That fallback is better than an error, but worse than solving the task.

The real problem is not just how to rank those outcomes. A scalar reward can encode success over fallback over error, yet its expected value can become hard to optimize: once a policy mostly falls back or mostly errors, the gradient that should move it toward success can shrink. How do you prefer fallback to failure without losing the pressure to improve?

The Core Move

GracefulRL maximizes a weighted sum of the log-probability of success and the log-probability of avoiding error:

$J=(1-c)\log p_S+c\log(1-p_E),\qquad 0<c<1.$

Here $p_S$ is success probability, $p_E$ is error probability, and $c$ controls the relative emphasis on avoiding errors. The first term pushes toward success; the second treats both success and fallback as preferable to error.

Quiz

In GracefulRL, what does the term log(1 − p_error) encourage?

It encourages probability mass on either success or fallback rather than error.

It rewards fallback while penalizing success.

It makes the objective depend only on success probability.

It forces the policy to choose a deterministic outcome immediately.

$ check

Why the Outcomes Need an Ordering

Now the version for readers who want more of the machinery: the three outcomes form a probability simplex, and the objective encodes an ordering by placing separate pressure on success and non-error probability.

Let $p_F$ be fallback probability, so $p_S+p_F+p_E=1$. Then $1-p_E=p_S+p_F$: the non-error event includes success and fallback. When $c=0$, the objective is just $\log p_S$, so fallback and error receive no distinct protection. When $c=1$, it is $\log(p_S+p_F)$, so success and fallback count alike as long as error is avoided. Between those endpoints, the objective combines both preferences. This gives the intended ordering—success over fallback over error—without pretending fallback is success.

Why the Log Terms Resist Saturation

Here is the part that gets technical: the log terms act as barriers near two undesirable boundaries, $p_S\to0$ and $p_E\to1$.

For outcome probabilities produced by a softmax, the log-success term has a direct gradient signal whenever success is rare, while the log-non-error term has a signal when error dominates. Under the stated regularity and non-degeneracy assumptions, the gradient remains bounded away from zero near either suboptimal outcome, fallback or error. By contrast, the expected-reward policy-gradient objective can have a vanishing gradient as probability concentrates on a single outcome. This is the recovery property: near fallback, the objective still supplies pressure toward success; near error, it also rewards shifting probability out of error.

The guarantee has conditions: it is stated for softmax-parameterized policies under regularity assumptions, not as a claim that every parameterization or training setup avoids every optimization failure. What happens when rare outcomes make rollout estimates unstable?

Why Rollout Weights Need Smoothing

Now the implementation detail that connects the objective to sampled trajectories: GracefulRL uses a rollout-based estimator with inverse-frequency weighting for the relevant outcome groups.

The algorithm specializes the TailRL objective to ternary rewards and adds a smoothing constant $\epsilon>0$. Smoothing shrinks large inverse-frequency weights when success or non-error trajectories are rare. That can reduce variance, at the cost of some bias; it does not change the conceptual target of preserving both success pressure and error avoidance. So the estimator makes the objective usable with finite rollouts, but introduces a bias–variance knob rather than a free improvement.

Minimal Causal Code

This loop shows the outcome ordering and the log-barrier objective directly. It assumes p_success and p_error are differentiable probabilities from the policy; a production rollout estimator additionally needs sampled-outcome weighting and smoothing.

import torch

c = 0.3

# Probabilities are produced by a differentiable policy/verifier surrogate.
p_success = policy_success_probability(batch)
p_error = policy_error_probability(batch)

# Avoid taking log(0) in finite-precision arithmetic.
loss = -((1.0 - c) * torch.log(p_success.clamp_min(1e-8))
         + c * torch.log((1.0 - p_error).clamp_min(1e-8)))

optimizer.zero_grad()
loss.mean().backward()
optimizer.step()

What You Gain, and What You Lose

GracefulRL expresses success over fallback over error while retaining a recovery signal near saturated fallback or error policies, under its stated assumptions. That matters when fallback is a legitimate safe outcome rather than a failure to be punished like any other.

The tradeoff is that the objective is a surrogate, not simply the expected value of a ternary reward. Its balance depends on $c$, and its sampled estimator uses smoothing to control large inverse-frequency weights, exchanging some bias for reduced variance. It also needs outcomes that can be distinguished by a verifier or environment. Fallback is only meaningful when the task has a real, defined fallback action; the objective cannot invent one.

Quiz

Why add a smoothing constant to the rollout-based GracefulRL estimator?

To remove the fallback outcome from the objective.

To guarantee zero bias in finite-rollout estimates.

To replace the log-probability objective with expected reward.

To shrink large inverse-frequency weights for rare outcome groups, trading some bias for lower variance.

$ check