IC-809Sparse dictionary features in Pythia-410m enable more precise causal localisation of indirect object identification behaviour than PCA, requiring fewer patches and smaller edit magnitudes for the same KL divergence
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, Lee Sharkey
Using activation patching on the IOI task, the authors compare interventions along sparse autoencoder dictionary features versus PCA components in Pythia-410m's residual stream (layer 11, r=4). They find that patching a smaller number of sparse features achieves the same KL divergence from the target output as patching more PCA components, and that the edit magnitude per feature is smaller. A non-sparse dictionary (alpha=0) does not show this advantage. The feature selection uses the ACDC algorithm applied to the dictionary features as a flat computational graph, averaged over 50 IOI test points. Results on other layers (3, 7, 11, 15, 19, 23) in Appendix F show the same pattern.
Evidence
interventional
Caveat
The dictionary is trained in a task-agnostic way; the authors note that generalisation to other tasks is expected but unconfirmed. The non-sparse (alpha=0) dictionary does not show the same precision advantage, and larger alpha values increase the minimum achievable KL divergence due to lower reconstruction accuracy.