Chunked Feedback for Parallel Token Processing

Recurrent decoder feedback limits parallel processing across known tokens; RLT-2 holds feedback fixed within chunks and commits the final output at complete chunk boundaries.

The short version

What it is

Chunked feedback Keep the decoder’s feedback state unchanged while processing a group of known tokens, then use the group’s final output as the next state when the group is complete.
Chunk size The group size sets how often feedback is refreshed and how many positions can be processed together.
Causal continuity Positions still attend only to allowed earlier positions, and attention cache entries carry across chunk boundaries.

Why it matters

A known prefix can be processed with parallel work inside each chunk without giving up causal outputs or continuity across chunks. The choice of chunk size balances fewer sequential stages against less frequent feedback.

Before getting into the mechanics, here is the short version: RLT-2 groups known tokens into chunks, keeps the recurrent feedback state fixed while a chunk is processed, and commits the last position’s output only when the chunk is complete. That gives the decoder room to process positions together without letting a token see the future.

Sequential path: With token-by-token feedback, the next decoder step waits for the previous output to become state. For a known prefix, that creates a long chain even though the tokens are already available. The canonical example is prompt prefill: every prompt token is known, but a recurrent decoder still has to respect its feedback schedule. The intended change is to group that work, not to pretend the causal dependencies vanish.
Causal behavior: Parallel execution is only useful if position $t$ still depends on the prefix through $t$ , not on later positions. That is, each output remains a function of tokens up to its own position. In plain terms, the chunk can share its incoming feedback state, while causal masks keep each position’s attention from reading ahead. The equation’s meaning is simply a restriction on what each position may read; there is no need to change the causal rule to batch the work.
Feedback frequency: Think of the boundary state as a notebook carried from one chunk to the next. RLT-2 does not rewrite that notebook after every token; positions in the chunk use the same incoming state, and the final position supplies the next state when the chunk closes. This is like deciding what to carry to the next town, though unlike a real notebook the decoder also retains attention keys across boundaries. If the prompt ends halfway through a chunk, it keeps the earlier complete-boundary state for continuation.

So what does the chunk size actually buy? The mechanism’s schedule makes that concrete: for a prefix of length $T$ , the number of sequential decoder stages is $\lceil T/B\rceil L_D$ , with up to $\min(B,T)$ positions in each batched operation. In words, bigger chunks mean fewer sequential stages and more positions together; the total decoder block evaluations remain $T L_D$ . At $B=1$ , every token closes a chunk and the behavior is RLT-1. At the opposite limit, RLT-0 drops the feedback merge and its related parameters rather than repeatedly merging with a constant state.

I like that this exposes the actual knob: how often should the decoder refresh feedback? It is not free parallelism, because layers remain sequential and masks still matter. So the remaining question is where the useful balance lies between more work together and less frequent feedback.

Read the deep dive

Why People Wanted It

The problem is the sequential path. In a recurrent decoder, each new token can depend on a feedback state produced by the preceding token, so a known prefix is processed through a chain of decoder stages. The real question is whether feedback must be refreshed after every token, or whether a decoder can keep causal behavior while processing several known positions together.

The Core Move

RLT-2 holds the feedback state fixed across a chunk and uses the final position’s output as the next state only when that chunk is complete. The stated rule is: every output predicts the next token, but only the last output of a complete chunk becomes the next feedback state; no additional summary token is used.

Why Feedback Can Wait for a Boundary

Now the version for readers who want more of the machinery: each position in a chunk merges its own encoder output with the same preceding boundary state, then the chunk runs through the decoder with causal masks. That shared state depends only on tokens before the chunk; causal self-attention still restricts a position to keys at or before itself, and encoder memory is likewise prefix-restricted. Thus each output remains causal even though positions in a chunk are handled together at each layer.

The exact merge equation is not reproduced in the cited text, so there is no formula to add here. The operative constraint is that the boundary state is shared within the chunk, while the per-position gate and encoder output can differ. Decoder layers remain sequential, but positions within a layer can run together.

Why Partial Chunks Keep the Earlier State

Here is the part that gets technical: a partial final chunk does not commit its last output as the continuing feedback register. It keeps the state from the preceding complete boundary. This makes continuation independent of where the prompt ends inside that unfinished chunk. Meanwhile, causal attention can use retained keys from earlier chunks; neither the attention window nor encoder positions reset at a chunk boundary.

At chunk size one, every token closes a chunk, so RLT-2 reduces exactly to RLT-1. At the other limit, RLT-0 has no feedback boundary: it also removes the constant initial-state merge and its associated normalization, gate, and feedback projection, rather than retaining a fixed merge at every position.

Why Chunk Size Changes the Schedule

For a known prefix of length $T$ , the decoder needs $\lceil T/B\rceil L_D$ sequential block stages, with up to $\min(B,T)$ positions in each batched matrix operation. In words, increasing the chunk size reduces the number of sequential stages while allowing more positions to be processed together. RLT-1 takes $T L_D$ stages, and RLT-0 takes $L_D$ ; all three evaluate $T L_D$ decoder blocks and generate one token per step. The tradeoff is feedback frequency against known-token parallelism.

Quiz

In RLT-2, what happens to feedback state while processing a chunk?

It is updated after every position.

It is replaced by an average of all chunk outputs.

It is reset at every chunk boundary.

It is held fixed within the chunk and committed from the final output at a complete boundary.

Minimal Causal Code

This loop illustrates the boundary rule only; it does not implement the attention masks, caches, or learned merge. The state changes after a complete chunk, not after each position.

def chunk_outputs(chunks, state, decoder):
    for chunk, is_complete in chunks:
        # Every position sees the same incoming boundary state.
        outputs = decoder(chunk, feedback_state=state)

        # Only a complete chunk commits its final output.
        if is_complete:
            state = outputs[-1]

        yield outputs

What You Gain, and What You Lose

RLT-2 exposes parallel work among known positions inside a chunk without discarding causal masking or cross-chunk cache continuity. The layer stack still runs in sequence, and the positions within each layer must obey causal masks. Larger chunks reduce sequential decoder stages but make feedback less frequent; smaller chunks provide more frequent feedback and less within-chunk parallelism. RLT-0 removes the recurrent merge rather than merely choosing an especially large RLT-2 chunk.

The schedule is explicit: all three intervals evaluate the same number of decoder blocks for a known prefix, but arrange those evaluations into different numbers of sequential stages. Chunk size is therefore a modeling and scheduling choice, not a claim that the underlying decoder work disappears.

Quiz

What does increasing the RLT-2 chunk size trade?

It removes causal masks and reduces total decoder blocks evaluated.

It makes the decoder layers parallel with one another.

It reduces sequential stages and increases known-token parallelism, while making feedback less frequent.

It resets attention caches at chunk boundaries.

Core Takeaway

RLT-2 buys parallel work by letting feedback wait until a complete chunk boundary.