Activation patching / Activation replacement / Cross-model activation patching (CMap)

anchor

Replace an activation in one forward pass with the value it took in another and measure how the output changes, isolating which component carries a behaviour.

Findings

• IC-013Entity recognition directions regulate attention to entity tokens in attribute extraction heads in Gemma and Llama models. [primary]
• IC-052GPT-2 small's IOI circuit activations are linearly decomposable into features for the io, s, and pos attributes, with the l10h0 name mover's attention decomposing into sparse pairwise feature interactions [primary]
• IC-053In GPT-2 small's l10h0 name mover queries, the io attribute is encoded with higher-magnitude features than the s attribute, and both are causally relevant, but SAEs preferentially learn io features due to the magnitude asymmetry [primary]
• IC-087Answer symbol production in OLMo 7B Instruct, Llama 3.1 8B Instruct, and Qwen 2.5 1.5B Instruct is causally attributed to a few middle layers and specifically their multi-head self-attention mechanisms, with a sparse set of 1-4 attention heads per layer responsible [primary]
• IC-088OLMo 7B Instruct and Qwen 2.5 1.5B Instruct exhibit a two-stage process for unusual answer symbols, initially assigning non-negligible probability to expected symbols (a/b/c/d) before switching to the actual prompt symbols at a specific later layer [primary]
• IC-149Intervening in the shared representation space using the dominant language (English) predictably changes model outputs for other data types, demonstrating the space is causally used rather than a vestigial byproduct [primary]
• IC-166A 1-dimensional subspace in a single layer encodes the context-versus-prior decision in Llama-3.1-8B, Gemma-2 9B, and Mistral-v0.3 7B, and setting this subspace steers the released (non-fine-tuned) models' behavior [primary]
• IC-218LLaMA3-8B and other LLMs solve arithmetic via a bag of independent heuristic neurons in middle and late MLP layers rather than a robust algorithm [primary]
• IC-220In Pythia-6.9B, the bag-of-heuristics mechanism emerges gradually during training and is the primary arithmetic mechanism from the earliest checkpoint showing good performance (23k steps) [primary]
• IC-359Mistral 7B and Llama 3 8B causally use circular subspaces to compute modular arithmetic on days of the week and months of the year [primary]
• IC-394Text generation in SDXL, DeepFloyd IF, and SD3 is controlled by less than 1% of parameters concentrated in specific cross- or joint-attention layers, and these layers are specialised for text content rather than visual template [primary]
• IC-722Entity tracking in Llama-7b is implemented by detecting and transmitting the positional information of the correct entity, with distinct head groups for position detection, transmission, and value fetching [primary]
• IC-723The entity tracking performance gap between Goat-7b and Llama-7b is primarily attributable to enhanced positional information in the value fetcher and position transmitter heads [primary]
• IC-745Function vectors for simple list-oriented tasks can be algebraically combined via addition and subtraction to produce new vectors that trigger composed tasks, sometimes outperforming 10-shot ICL. [primary]
• IC-809Sparse dictionary features in Pythia-410m enable more precise causal localisation of indirect object identification behaviour than PCA, requiring fewer patches and smaller edit magnitudes for the same KL divergence [primary]
• IC-840GPT-2 Small's name mover heads exhibit disrupted attention patterns under out-of-distribution Gaussian noise corruption [primary]