IC-7951D subspaces of MLP activations found by DAS in GPT-2 Small (IOI) and GPT-2 XL (factual recall) produce apparent causal effects that are interpretability illusions driven by causally disconnected components activating dormant pathways
Aleksandar Makelov, Georg Lange, Atticus Geiger, Neel Nanda
The paper demonstrates that subspace activation patching along 1D directions found by DAS in MLP hidden activations yields a strong end-to-end causal effect (flipping IOI predictions or changing recalled facts), but this effect is an illusion. Decomposing the direction into its nullspace (causally disconnected) and rowspace (causally relevant) components of the MLP output projection shows that the nullspace component carries the input variation while the rowspace component is dormant. Removing the nullspace component reduces the IOI FLDd from 46.7% to 13.5%, and in factual recall the effect disappears entirely. In contrast, patching in the residual stream (v_resid, v_grad) produces faithful effects that persist after removing the nullspace component, confirming the illusion is specific to non-bottleneck components.
The illusion is demonstrated in two specific settings (IOI and factual recall) with specific model sizes; the authors argue it should be prevalent but do not prove it universally. The IOI effect for v_mlp is relatively small in absolute terms (4.2% interchange accuracy) though significant for a single component.