IC-296The degree to which SAE features are active at multiple residual-stream layers increases with model size in Pythia, Gemma 2, Llama 3.2, and GPT-2
Tim Lawson, Lucy Farnik, Conor Houghton, Laurence Aitchison
Using a multi-layer SAE trained on the residual stream activations of each model, the authors measure how SAE latents are distributed across layers. When aggregating over 10 million tokens, the fraction of total variance explained by individual latents (i.e., the degree of multi-layer activity) ranges from 54% to 88% and broadly increases with model size. In contrast, for a single token the fraction of variance explained by individual tokens is only 1–10%, meaning that for a given prompt most latents fire at a single layer. This is consistent with the mean cosine similarity between adjacent-layer residual stream vectors increasing with model size (Figure 1), indicating that the residual stream preserves information more faithfully across layers in larger models.
Evidence
correlational
Key metric
fraction of total variance explained by individual latents (eq. 11): between 54 and 88%, increasing with model size; fraction of variance for an individual latent explained by individual tokens (eq. 12): on the order of 1 to 10%; variance of layer distributions about two orders of magnitude greater when aggregating over tokens vs. a single token
Caveat
The multi-layer SAE is a new instrument introduced by this paper; the observed multi-layer structure depends on the SAE's hyperparameters (expansion factor r=64, sparsity k=32) and on the choice of residual stream basis (no tuned lens applied). The authors note that the last-layer activations are excluded from the analysis.