Turn-Boundary Skip-Observation GAE

Why People Wanted It

An agent may generate many tokens, receive an observation, then generate more tokens before the task ends. Standard GAE can treat observation positions as recursion steps even though the policy made no decision there. The problem is not just sequence length: observation tokens can make credit fade without a corresponding action. So the real question was whether the credit clock should follow text positions or interaction turns—and how to split the calculation without changing it.

The Core Move

TB-GAE places its discount and trace-decay clock on generated positions and turn boundaries, while observations condition values but are not recursion nodes. For generated position $j$, use its value $v_j$, causally assigned reward $\tilde r_j$, and termination marker $d_j$; the paper specifies the clock and recursion but the supplied page text does not preserve their displayed formulas legibly, so no missing algebra is reconstructed here.

Why Observation Tokens Should Not Advance the Credit Clock

Now the version for readers who want more of the machinery: the key distinction is between a position at which the policy chooses an action and an observation that merely changes what the next action can depend on. TB-GAE links generated positions $u_j$ to pre-token values $v_j=V(h_{u_j})$, turn IDs $t(j)$, and causally assigned rewards $\tilde r_j$. Observation nodes can affect those values, but do not enter the advantage recursion.

The turn clock uses $0\leq\gamma_{\mathrm{turn}},\lambda_{\mathrm{turn}}\leq1$. The boundary factors are $\Gamma_j$ and $\Lambda_j$, and the combined recursion weight is $w_j=\Gamma_j\Lambda_j(1-d_j)$. In words: discount and trace decay are applied at the relevant turn boundary, while true termination shuts off the successor trace. At termination, successor value and trace are zero; the clock factors are set to one rather than querying a nonexistent next turn. This makes attenuation across $B$ turn boundaries $(\gamma_{\mathrm{turn}}\lambda_{\mathrm{turn}})^B$ for fixed retained inputs, independent of observation length or token count within a turn. It also changes the discounted objective. What does that mean when feedback arrives only at the end?

Why Sparse Feedback Still Belongs to Its Causal Position

Here is the part that gets technical: a terminal reward must remain attached to its causal generated position, not be copied to every chunk or observation. If a turn receives reward only at its final generated token, telescoping within that turn yields prefix-specific advantages: positions can share a return while having different values and therefore different advantages. The terminal successor terms vanish, and each reward occurs once.

With unit clocks, the advantages reduce to the sum of subsequent assigned rewards minus the current value on complete episodes, matching standard GAE only when inputs, values, reward placement, and endpoints agree. This is not a license to move reward around: causal placement and multiplicity are part of the estimator. How can a long reverse scan be partitioned without changing that result?

Why Exact Chunking Needs Both Successor Quantities

Now the exact continuation condition: at a computational boundary $b$, retaining the value bootstrap but resetting the trace loses the contribution of the future trace. Exact continuation requires both $v_{b+1}$ and $A_{b+1}$, or an unsplit reverse scan. A task-defined terminal budget sets $d=1$; a sampling cut or computational partition does not automatically end the task. If the future trace is unknown, a cutoff is approximate.

The condition is fixed retained inputs. A summary changes conditioning, so transporting state across it does not establish equivalence to the original fixed-input continuation. Where, then, is the practical tradeoff between cleaner turn-level decay and the objective you intend to optimize?

Minimal Causal Code

This transparent sketch shows the reverse recurrence over generated positions only. clock(j) supplies the boundary factors described above; it is not an implementation of observation-token steps.

# Inputs are ordered generated positions; observations are not entries here.
advantages = [0.0] * len(rewards)
next_advantage = 0.0
next_value = 0.0

for j in reversed(range(len(rewards))):
    if done[j]:
        next_value = 0.0
        next_advantage = 0.0
        gamma_factor, lambda_factor = 1.0, 1.0
    else:
        gamma_factor, lambda_factor = clock(j)

    delta = rewards[j] + gamma_factor * next_value - values[j]
    weight = gamma_factor * lambda_factor * (1.0 - float(done[j]))
    advantages[j] = delta + weight * next_advantage
    next_value = values[j]
    next_advantage = advantages[j]

For chunked execution, the reverse scan must receive both successor value and successor advantage at the boundary. Resetting only the latter changes the result.

What You Gain, and What You Lose

Turn-boundary decay avoids making observation length itself the source of extra trace attenuation, and exact chunk continuation can preserve the unsplit calculation when retained inputs stay fixed. The costs are explicit: changing the clock changes the discounted objective; a task boundary must be distinguished from a compute or sampling cut; and unknown future traces force an approximation. Nor does the construction imply a universal variance reduction or policy-invariance guarantee. The choice is therefore about which credit clock matches the interaction and objective—not simply about making GAE faster.