Contrastive Covariance Subspaces for Activation Steering

Alexandros Xenos, Manos Plitsis, Niki Foteinopoulou, Yannis Panagakis, Ioannis Patras

arXiv 2610.14025 36 pages Read the paper

The short version

The problem: Adding a direction to a language model’s internal activity can encourage a behavior, such as writing more cheerfully. But the direction may also contain unrelated patterns from the examples used to make it, so steering can introduce irrelevant content or make responses worse.
The idea: The paper’s method, Contrastive Covariance Steering, compares the target behavior with the average of all available behavior classes, then identifies a compact set of directions where the target varies more than the alternatives. It filters the steering direction through that set—for example, keeping the signal for a desired tone while removing a topic pattern shared with other example groups.
What it got: Across three language models and four test areas, it beat every competing activation-steering method in all 12 comparisons. On one benchmark, its score was 48–70% better than the strongest competing steering method.

Module from this paper

Contrastive Covariance Subspaces for Activation Steering

A plain target-versus-neutral direction can carry nuisance variation, so Contrastive Covariance Steering builds a target-specific subspace that preserves target scatter while suppressing competing-class scatter, then projects the target’s grand-mean displacement into it.

activation-steeringrepresentation-geometryeigendecomposition