Pair-Targeted Kronecker Delta Memory Updates

Why People Wanted It

A matrix memory can store key–value associations by adding each pair’s outer product. But additive writes do not revise an association; later writes can bury it. The ordinary Delta Rule does revise, yet its correction depends on the memory’s current retrieval, so a write can disturb other associations whose keys overlap. The real question is whether you can correct the score of one chosen pair without letting the memory’s other contents choose the direction of that correction.

The Core Move

Correct the scalar score for the presented key–value pair, and move the memory only along that pair’s outer product:

$S_{t+1}=S_t+\beta_t(c_t-v_t^\top S_tk_t)v_tk_t^\top.$

Here $v_t^\top S_tk_t$ is the pair’s current score, $c_t$ is its target, and $\beta_t$ controls the correction size. The update is rank one: the pair itself supplies the direction, while the scalar residual says how far to move.

Quiz

What determines the direction of a Kronecker Delta Rule memory update?

The difference between the entire retrieved vector and the target vector

A fixed direction independent of the presented pair

The key overlap with every previously stored association

The presented value–key outer product, scaled by the scalar score error

$ check

Why the Scalar Score Changes the Correction

Here is the part that gets technical: the key distinction is between correcting a whole retrieved value and correcting one scalar association. A matrix memory $S$ maps a key $k$ to a value-like vector $Sk$. The ordinary Delta Rule compares that vector with the desired value, using residual $Sk-v$. The Kronecker Delta Rule instead asks for the scalar bilinear score $v^\top Sk$ and compares it with scalar target $c$.

Write $a=k\otimes v=\operatorname{vec}(vk^\top)$ and $s=\operatorname{vec}(S)$. Then $v^\top Sk=a^\top s$, so the update becomes $s_{t+1}=s_t+\beta_t(c_t-a_t^\top s_t)a_t$. In words, this is a scalar error-correction step along the feature vector belonging to the pair. The target sets the desired score; the memory’s other entries do not set the direction.

For a repeated pair, the scalar error contracts by the factor $1-\beta_t\|a_t\|^2$. Thus a positive step size below $2/\|a_t\|^2$ contracts the error, and the step size $1/\|a_t\|^2$ reaches the target in one update. What does making the direction pair-specific buy when other associations are present?

Why the Rank-One Direction Limits Interference

Now the version for readers who want more of the machinery: the update matrix is always a scalar multiple of $vk^\top$, so it changes $S$ only in that rank-one direction. For a test pair $(k',v')$, the change in its read-out is proportional to $(v'^\top v)(k^\top k')$ times the current pair’s scalar residual. In words, both the key overlap and the value overlap must be present for this update to alter that other pair’s score.

That gives an exact protection condition: if either $k'\perp k$ or $v'\perp v$, the test pair’s bilinear read-out is unchanged. This is more selective than interference controlled by key overlap alone. It is not universal isolation: non-orthogonal pairs can still interact, and an inaccurate current score leaves a residual that can produce that interaction. So what if the goal is not merely to revise a score, but to erase it?

Why the Target Enables Erasure

Here is the deeper consequence: setting $c=0$ asks the update to drive the presented pair’s scalar score to zero, rather than to store a particular output vector. For the same pair, this is the same contraction of its scalar error toward the chosen target; with the one-step size $1/\|k\otimes v\|^2$, the score reaches that target in one update.

The update can also be split into an erase term and a write term by expanding the residual: $S_{t+1}=S_t-\beta_t(v_t^\top S_tk_t)v_tk_t^\top+\beta_tc_tv_tk_t^\top$. The first term subtracts the pair’s current contribution along its direction; the second adds the target-scaled pair direction. This is targeted erasure of a bilinear association, not a guarantee that every other non-orthogonal association is untouched. How much protection and capacity do you get from this pair-addressed view?

Minimal Causal Code

The loop below exposes the state change directly; it is intentionally not optimized.

import torch

def kronecker_delta(memory, keys, values, targets, beta):
    # memory: [value_dim, key_dim]
    for key, value, target, step_size in zip(keys, values, targets, beta):
        current_score = value @ memory @ key
        residual = target - current_score
        memory = memory + step_size * residual * torch.outer(value, key)
    return memory

Each iteration measures one pair’s score, forms its scalar error, and adds only the pair’s outer-product direction to memory.

What You Gain, and What You Lose

You gain a revision rule whose correction direction is determined by the presented pair, scalar error correction for that pair, and doubly gated interference: both key and value overlap matter. You can set the target to zero to erase that pair’s score. In a vectorized view, each pair imposes one scalar linear constraint on the memory, so independent pair features can represent scalar associations.

You do not get perfect isolation for arbitrary pairs: overlapping keys and values can still cause crosstalk. Nor is this the same objective as retrieving a whole value vector; it targets the scalar bilinear association. Step size still matters, and this rule by itself says nothing about a complete sequence-mixer architecture or its execution strategy.

Quiz

When is another pair’s bilinear read-out exactly protected from a Kronecker Delta update?

When either its key or its value is orthogonal to the updated pair’s corresponding vector

Whenever its key differs from the updated key, even if they overlap

Only when both its key and value are identical to the updated pair

For all other pairs, regardless of overlap

$ check