Stable Expected Dynamics of Kronecker Delta Memory

Why People Wanted It

A memory that receives one randomly sampled association at a time can keep moving even when every association cannot be satisfied simultaneously. That makes a useful question more precise than “does the memory converge?”: does its average find a meaningful solution, and do the random fluctuations stay under control? For long contexts, stability means understanding both the mean and the distribution of the state—not demanding that every individual trajectory land on one exact memory.

The Core Move

Treat each association as a stochastic least-squares update: the expected state follows a deterministic linear recursion toward the least-squares memory, while the random iterates form a stable Markov chain under the stated step-size condition.

For the vectorized memory $s$, the exact expected recursion is

$$\mathbb{E}[s_{t+1}] = (I-\beta\bar{A})\mathbb{E}[s_t]+\beta\bar{b}.$$

Its fixed point is $s^{LS}$, the least-squares memory. The random state need not converge to that point; instead, it converges in distribution to a stationary law whose mean is exactly $s^{LS}$.

Quiz

When random updates keep fluctuating because associations are inconsistent, what is the convergence guarantee described here?

Every individual memory trajectory converges to the least-squares memory.

The stationary distribution exists only when all associations are consistent.

The expected iterate converges to the least-squares memory, and the random state converges in distribution to a stationary law with that mean.

The mean diverges, but the state becomes deterministic.

$ check

Why the Mean Can Converge When Individual Memories Do Not

Now the version for readers who want more of the machinery: the key distinction is between averaging the update and asking a single random trajectory to stop moving. In an over-capacity setting, associations can be inconsistent, so the stochastic updates continue to respond to whichever association was sampled. The mean update, however, averages those sample-wise corrections into a linear recursion.

The paper defines the least-squares memory by minimizing the expected squared prediction error, $\mathbb{E}[(v^\top S k-c)^2]$. In words, it is the memory with the smallest average squared mismatch over the input-association distribution. With the vectorized feature $a=k\otimes v$, the prediction is $a^\top s$; the paper’s expected recursion has the least-squares state as its unique fixed point. Under the stated admissible step-size condition, the mean error contracts geometrically. This is a claim about the expectation, not pointwise convergence of each random iterate. So what replaces convergence of an individual memory?

Why the Random State Has a Stationary Distribution

Here is the part that gets technical: each random update defines a transition of a time-homogeneous Markov chain on the vectorized memory space. The stated contraction on probability laws with finite second moment gives a unique stationary distribution, and the law of the iterates converges to it geometrically from any initialization. Its stationary mean is exactly $s^{LS}$.

That separates two kinds of stability. The distribution settles even though a realized memory may keep fluctuating; and the mean of that settled distribution has no offset from the least-squares solution. The source also describes a Wasserstein-2 synchronous-coupling contraction in its appendix analysis, but the main result needed here is the existence, uniqueness, and geometric convergence of the stationary law. What controls whether the size of those fluctuations stays bounded?

Why the Expected Memory Norm Stays Bounded

Now the norm-bound layer: the analysis bounds the least-squares memory using explained target energy, with residual energy only shrinking that bound. In the non-expansive regime $\kappa\leq 1$, it further bounds the steady state and, when initialized at zero, every iterate; the transient factor decays geometrically. The paper states that from the first step onward the expected memory is bounded by a constant determined by the data distribution alone.

This is an expected-memory stability result, not a promise that every possible sample path has a fixed maximum norm. It is useful precisely because a random memory can keep changing while its expected size remains controlled. How should you read that guarantee in practice?

Minimal Causal Code

The loop below makes the sampled update visible. It illustrates the stochastic least-squares step; it does not implement the paper’s separate norm-bound proof.

import torch

def run_delta_memory(features, targets, beta, steps, s0):
    # features[t] is the vectorized association feature a_i.
    # targets[t] is its scalar target c_i.
    s = s0.clone()
    states = []

    for t in range(steps):
        a = features[t]
        c = targets[t]
        residual = a @ s - c
        s = s - beta * residual * a
        states.append(s.clone())

    return s, torch.stack(states)

What You Gain, and What You Lose

You gain a precise stability picture for random associations: the expected memory targets the least-squares solution, the stochastic dynamics approach a unique stationary distribution, and expected memory norms are bounded under the stated conditions. You do not gain pointwise convergence of every iterate to one memory in the inconsistent case; the stationary state can retain sampling-driven fluctuations. The guarantees also depend on the assumptions, including random sampling and an admissible fixed step size. This module is about those dynamics, not the separate gated sequence-mixer architecture or its chunkwise execution.