Contrastive Covariance Subspaces for Activation Steering

Why People Wanted It

A difference between a target class and one neutral class can point in the right broad direction while also carrying topic, style, or other variation that the dataset happened to associate with the target. The problem is not only where the target mean sits; it is which directions vary for the target and which of those directions also vary for competing classes. So the real question was never just how to subtract a baseline. It was how to keep the target’s displacement while filtering out directions that look shared or nuisance-like across the alternatives.

The Core Move

Contrastive Covariance Steering (CCS) constructs a target-specific subspace by rewarding target scatter and penalizing average non-target scatter, then projects the target’s displacement from the grand mean into that subspace:

$A_t = S_t - rS_{N_t}, \qquad v_{\mathrm{CCS}} = P_t(\mu_t-\mu_G).$

Here $S_{N_t}$ is the average scatter of the competing classes, $r\geq 0$ controls its penalty, and $P_t$ projects onto the leading eigenspace of $A_t$.

Quiz

What does CCS do after computing the target’s displacement from the grand mean?

Finds the subspace by gradient descent over steering vectors.

Projects it into a subspace that rewards target scatter and penalizes average nuisance scatter.

Adds every competing class mean to the target displacement.

Uses only the within-class covariance of a single neutral class.

$ check

Why the Grand Mean Gives a Different Displacement

Here is the deeper version of the centering choice: CCS measures the target from the average of all class means, not from one chosen neutral class. For classes $c\in\mathcal C$, define

$\mu_G = \frac{1}{|\mathcal C|}\sum_{c\in\mathcal C}\mu_c, \qquad \delta_t=\mu_t-\mu_G.$

So the reference is the center of the class means, and the target displacement is its offset from that center. This changes the baseline rather than the subspace: a single neutral reference can make the first-order contrast depend on that particular class, whereas the grand mean uses all class means. The activations are centered on $\mu_G$ when class scatter is computed, so the scatter includes both within-class variation and the class mean’s displacement from the common center.

Why the Scatter Difference Suppresses Shared Directions

Now the version for readers who want more of the machinery: the contrast is between projected energy, not merely a comparison of covariance matrices. Let $H'_c$ contain class-$c$ activations centered on $\mu_G$, scaled by $n_c^{-1/2}$. For a candidate orthonormal basis $W$, $\|W^\top H'_c\|_F^2$ is the class’s mean squared projected norm, with the scaling making it independent of class size. The target-specific objective is

$\max_{W^\top W=I_k}\;\|W^\top H'_t\|_F^2-r\,\frac{1}{|N_t|}\sum_{c\in N_t}\|W^\top H'_c\|_F^2.$

In words, choose a $k$-dimensional subspace where target activations have large projected energy and the typical competing class has little. Because the centering is at the grand mean, a competing class is penalized both for varying along a direction and for sitting away from the shared center along it. The knob $r$ sets how strongly that average nuisance energy counts. This leaves a practical question: how do you find the best subspace without searching over every possible basis?

Why One Eigensystem Produces the Steering Vector

Here is the part that gets technical: the projected-energy objective can be written as a trace, $\operatorname{tr}(W^\top A_tW)$, where $A_t=S_t-rS_{N_t}$. Its maximizing orthonormal columns are the $k$ leading eigenvectors of the symmetric matrix $A_t$; placing them in $W_t$ gives the projector $P_t=W_tW_t^\top$. Finally, $v_{\mathrm{CCS}}=P_t\delta_t$: keep only the part of the grand-mean-centered target displacement that lies in the target-specific contrastive subspace. The remaining question is what this selective filter buys—and what it cannot promise.

Minimal Causal Code

This deliberately direct sketch makes the state—the current contrastive matrix and selected subspace—visible. It assumes you already have class means and grand-mean-centered activation matrices with the stated class-size scaling.

import torch

# centered[c]: [d, n_c], columns scaled by n_c**(-1/2)
# means[c]: class mean; target and nuisance labels are already chosen
S = {}
for c in classes:
    S[c] = centered[c] @ centered[c].T

S_nuisance = sum(S[c] for c in nuisance) / len(nuisance)
A = S[target] - r * S_nuisance

# A is symmetric; take the eigenvectors with largest eigenvalues.
eigenvalues, eigenvectors = torch.linalg.eigh(A)
W = eigenvectors[:, -k:]
P = W @ W.T
mu_grand = torch.stack([means[c] for c in classes]).mean(dim=0)
delta = means[target] - mu_grand
v_ccs = P @ delta

What You Gain, and What You Lose

CCS stays in the closed-form additive-steering setting, but uses second-order information from multiple competing concepts to filter a mean displacement. The eigendecomposition avoids gradient-based subspace optimization. In return, you must choose the target, nuisance classes, subspace dimension $k$, and contrastive weight $r$; the resulting filter is specific to that target and to the classes represented in the data. Suppressing average nuisance scatter is not a guarantee that every nuisance behavior is removed: class overlap and dataset imbalance can leave shared or incidental structure in the retained direction. The steering vector is a geometric filter, not a proof of behavioral independence.

Quiz

Which limitation follows from CCS using average nuisance scatter to shape its subspace?

It requires a separate neutral reference class for every token.

It does not guarantee removal of every nuisance behavior, especially when classes overlap or data are imbalanced.

Its steering vector is not additive.

It cannot use second-order information from competing classes.

$ check