Persistent Working Memory for Multi-Turn Hybrid Context
Hybrid turns mix text, latent thoughts, and tool observations that cannot be reconstructed from text alone; the agent preserves their accumulated key-value context and resumes from it across turns.
Read the paper ↗What it is
- Cross-turn working memory The agent keeps the model context built from text, internal thoughts, and tool observations, then continues the next turn from it.
- Lossless continuation With the same model and inference setup, the retained context can support continuation without rebuilding the earlier conversation as text.
Why it matters
Hybrid histories contain more than ordinary words. Keeping the accumulated context lets the agent continue across turns without translating its internal thoughts into a text transcript.
Before getting into the machinery, here is the idea: instead of rewriting every previous turn into a new text prompt, the agent keeps the context the model has already built and carries it into the next turn. That matters because a hybrid conversation includes internal latent steps and tool observations, not just words you can paste back in.
-
Representation: A normal text-and-tool agent can append its transcript and encode it again. A hybrid agent also has latent vectors, which ordinary tokenization cannot reconstruct as the same hybrid history. The canonical example is a tool call: the conversation has the tool request and response, but the working context can also include latent steps around them. The memory keeps the accumulated model context rather than turning everything into a written transcript.
-
Continuity: Think of working memory as a notebook carried from one turn to the next: you add what happened and pick up where you left off. Unlike a human notebook, its entries are model key-value context, not readable notes; the analogy is only about carrying state forward. The memory is initialized from the input, then grows as text, latent thoughts, and tool observations are processed. The useful intuition is that the agent doesn't need to make an old latent step pretend to be a text token just to continue.
So what exactly is being carried? The model collects key-value entries across its layers while processing those different kinds of input. They share an input space, so the Transformer can process them into one continuous working memory. In practical terms, this is closer to retaining the model's context cache than to saving a transcript. What the paper calls lossless continuation is conditional: with the same model and inference settings, complete memory, unchanged positions, and the same causal operations and masks, continuing from the cache matches processing the full prefix. That does not mean you can edit old turns arbitrarily and expect the cache to remain valid.
The important distinction is between preserving context and reconstructing it. The agent resumes with accumulated key-value entries and appends new ones as it goes; it does not need to convert earlier latent thoughts into text. One related mechanism handles switching between internal reasoning and text interaction, but the point here is simply what happens to context across turns.
So the tradeoff is straightforward: you avoid losing the hybrid representation in a text rewrite, but continuation relies on keeping the complete memory and consistent inference conditions. When the next turn starts, is the agent continuing the same context, or rebuilding a different one from a transcript?
Read the deep dive
Why People Wanted It
A conventional agent can append its previous text and tool output to the next prompt, then encode that prefix again. But a hybrid trajectory also contains latent vectors: reconstructing the history as ordinary text would not preserve those vectors in their original form. The real question is not merely how to continue a conversation, but how to carry its complete hybrid context into the next turn without turning it into a different representation.
The Core Move
Keep the Transformer’s accumulated key-value entries as working memory, and continue generation from that memory on the next turn. The paper gives no displayed update formula for this process; its account is operational: initialize memory from the input, append generated entries as the trajectory unfolds, and resume from the resulting memory.
Why the Memory Holds More Than a Text Transcript
Here is the technical distinction: the working memory collects key-value entries across all layers as the model processes text, latent thoughts, and tool interactions. Those contents are mapped into a shared input space through the input embedding layer or HybridLink, so the Transformer can process them and collect their entries in the same memory. This is not a transcript stored as words; it is the accumulated model context.
Why Resuming Can Preserve the Earlier Context
Now the deeper continuation argument: with the model and HybridLink fixed, and inference settings kept consistent, the appendix states that continuing with complete working memory gives the same hidden output as processing the full prefix. The stated conditions include using the same deterministic causal operations, masks, and positional encoding, and preserving existing key-value entries. Intuitively, causality means appending a new input does not change the earlier entries; with the same positions and operations, the cached run can continue the computation rather than reconstruct earlier turns. This is a conditional claim about continuation under those settings, not a claim that arbitrary edits to past context can be handled by the cache.
Minimal Causal Code
The following sketch makes the memory lifecycle visible; model.step stands for a causal forward step that returns new key-value entries. The quoted pages describe the process but do not specify a runnable API.
memory = None
for item in hybrid_inputs: # text, latent input, or tool observation
output, new_entries = model.step(item, memory=memory)
if memory is None:
memory = new_entries
else:
memory = append_entries(memory, new_entries)
What You Gain, and What You Lose
You retain text, latent thoughts, and tool observations in their accumulated context, and can continue across turns without reconstructing earlier turns in token space. The tradeoff is that continuation depends on preserving the complete working memory and using consistent inference settings; it is not equivalent to freely rebuilding or editing the conversation as text. The account here concerns context continuity, not a guarantee that every hybrid agent uses the same memory interface.
A hybrid agent can carry its history forward as accumulated model context instead of rewriting that history as text.