Agent Decomposition for Bounding Peak Inference Memory

Why People Wanted It

The problem is the peak. A long-context call must keep its active key–value cache in memory while it processes its prompt and generates output; that cache grows with context, and systems must provision for the largest call, not the average. So the real question is not only how to choose useful evidence, but how to keep any one call from holding all of it.

Retrieval and decomposition can both appear to make a task more manageable, but they change different things. Retrieval changes which evidence enters a call. Decomposition changes how many calls share the work. If the largest call is still the memory bottleneck, only the latter directly bounds its active working set.

The Core Move

Split the task across cooperating agents, each handling part of the evidence and passing a distilled result onward, so no single call has to hold the whole context. The source gives no general formula for this bound; it describes a measured peak as the largest single call’s prompt-plus-generation footprint.

Why Spreading Work Bounds the Active Cache

Now the version for readers who want more of the machinery: the key distinction is between the total evidence used across a task and the evidence active in one call. In a single-pass setup, one call receives the full context, so its cache reflects that call’s prompt and generation. In a decomposed setup, agents handle parts of the evidence and pass distilled state; the active cache belongs to each call in turn, rather than one call containing the entire task context. The architecture therefore bounds the per-call working set as total evidence grows, without claiming that total work or total cache traffic disappears.

The system described separates an ephemeral per-call KV cache (L1) from a key–value store of typed Communication Units (L2). L2 lets a later agent receive a distilled predecessor output rather than raw context. The important constraint is that this is a bound on the active per-call cache, not a claim that all evidence or intermediate state occupies no memory.

Why Retrieval Does Not Bound the Largest Call

Here is the part that gets technical: retrieval filtering and decomposition act on different axes. Retrieval changes which evidence enters a call; if the selected passages are concatenated into one generation call, that call still has to process the concatenated prompt. Decomposition instead spreads evidence handling across calls and passes condensed intermediate results. That is why better filtering alone does not imply a smaller peak KV working set.

In the reported comparison, retrieval-augmented generation (RAG) shares the reasoner, critic, and evidence pool with the single-pass and decomposed arms, while the RAG passages are still concatenated into one generation call. The measured peak for that RAG baseline is indistinguishable from closed-book single-pass in the reported table; the decomposed system has a lower measured peak. This comparison supports a narrow systems claim: retrieval-matched filtering is not the same intervention as spreading the work across calls.

Why the Bound Is a Trade, Not a Free Reduction

For readers who want the final layer: the measured quantity is the largest single call’s prompt-plus-generated footprint, converted using the served model’s stated KV geometry. It is not total device memory and not cumulative KV traffic. In the reported setup, the decomposed system’s peak is lower than both baselines, while cumulative KV traffic is higher. The operational bargain is therefore lower peak working memory in exchange for more calls and more total traffic—not less computation in every sense.

Minimal Causal Code

# Each call sees a piece of the evidence, not the whole task context.
intermediate = []
for evidence_part in evidence_parts:
    result = agent_call(evidence_part)
    intermediate.append(condense(result))

# The final call receives condensed outputs from earlier calls.
answer = agent_call(intermediate)

The loop makes the causal move visible: divide the evidence, make separate calls, and pass condensed results forward. It is illustrative, not a reproduction of the system’s implementation.

What You Gain, and What You Lose

Decomposition can bound the active KV cache per call, which matters when peak cache memory is the constraint. Retrieval augmentation can select evidence, but when selected passages are fed into one generation call it does not, by itself, spread that call’s working set across calls. The cost of decomposition is that the task is handled through more calls; in the reported setup, cumulative KV traffic is higher even though peak KV is lower. Whether that trade is worthwhile depends on whether peak memory, rather than total traffic, is the binding resource.