Persistent Memory in Multi-Agent LLM Inference: What It Costs, What It Buys, and When You Can Tell

The short version

The problem

Long questions can use more temporary memory than a device can comfortably provide. Systems often split the work among agents and save past reasoning for reuse, but it can be difficult to tell whether that saved memory really helps later answers.

The idea

The paper measures these two choices separately: agents handle pieces of a task and pass along condensed results, while a separate store saves reasoning traces for possible reuse. It also explains when a test can fairly reveal a benefit from saved traces—for example, later questions must have something useful in common with earlier ones, and traces must not leak between test conditions.

What it got

Splitting the work lowered the largest temporary memory use to 14.3 mebibytes per question, compared with about 35 mebibytes for the comparison systems. Saved traces used a little more memory but showed no measurable accuracy improvement: the estimate was +0.015, with a 95% confidence interval from −0.011 to +0.046.