Depth-dependent structure

A property is organised progressively across the layers of a network rather than being present throughout: the structure is weak or absent early and sharper deeper in. Findings of this kind are claims about where in the model to look, and they require measuring at several depths.

Findings

• IC-003HyperDAS dynamically selects intervention tokens and learns linear subspaces in Llama3-8b that mediate entity attributes.
• IC-019Visual token representations in LLaVA-1.5 evolve to align with interpretable text tokens
• IC-020LLaVA-1.5 extracts object information directly from visual tokens to the last token in mid-late layers
• IC-022Safety layers in Llama-2-chat-7b show substantial divergence during VL adaptation, correlating with safety degradation
• IC-043Five ~7B decoder-only LLMs develop a high-intrinsic-dimensionality phase in intermediate layers that marks the transition from surface-form to abstract linguistic processing, with earlier onset predicting better next-token prediction
• IC-060SAE features in Pythia-160m and Mamba-130m exhibit high cross-architecture similarity with a depth-scaled correspondence
• IC-062Llama 3 70B implements temporal difference learning in-context for reward-based RL, with causally relevant SAE features in its residual stream, while Llama 3 8B performs at chance
• IC-063Llama 3 70B learns global graph structure via TD learning, building successor-representation-like geometry in its residual stream that is causally supported by TD latents
• IC-064The TD learning mechanism identified in Llama 3 70B generalizes to Gemma-2-27B and Qwen-2.5-72B across all three tasks
• IC-087Answer symbol production in OLMo 7B Instruct, Llama 3.1 8B Instruct, and Qwen 2.5 1.5B Instruct is causally attributed to a few middle layers and specifically their multi-head self-attention mechanisms, with a sparse set of 1-4 attention heads per layer responsible
• IC-088OLMo 7B Instruct and Qwen 2.5 1.5B Instruct exhibit a two-stage process for unusual answer symbols, initially assigning non-negligible probability to expected symbols (a/b/c/d) before switching to the actual prompt symbols at a specific later layer
• IC-1005Social bias neurons in BERT-base-cased and RoBERTa-base are concentrated in the deepest transformer layers
• IC-1015GPT-J and 10 other LLMs exhibit overthinking: calibrated accuracy given incorrect few-shot demonstrations peaks at a critical layer then declines, and ablating 5 false induction heads in late layers reduces the accuracy gap by 38.9% on average
• IC-1079Causal tracing reveals that mid-upper MLP layers (layers 18–25 in 7B) are the primary mediators of stereotypical gender bias in LLaMA, while the last layers show negative coefficients that counter the bias
• IC-1127The LM head in GPT-2, GPT-J, BLOOM, Pythia, and LLaMA-2 projects all input token hidden states into interpretable token distributions over the vocabulary, and these distributions converge approximately monotonically toward the final layer's distribution
• IC-117Hallucination heads in LLaVA-7B and MiniGPT-4 are concentrated in the middle and deeper layers of the transformer
• IC-1207For GPT-2-XL, backdoor injection via parameter editing is most effective on intermediate layers (15-35) and notably less effective on the first 10 and last 5 layers
• IC-122Concept representations in Llama-2-7B, Gemma-7B, and Llama-2-13B become more consistent in deeper layers
• IC-1227SAM's segmentation pretraining shifts attention heads toward local focus in deeper layers, unlike its MAE initialization which retains global attention throughout
• IC-130Amplifying visual-token attention in high layers (16-32) of released VLMs reduces sycophancy while preserving VQA accuracy, indicating that insufficient high-layer visual attention is a key cause of sycophancy
• IC-1317Llama-2 and Pythia models contain linear representations of space and time that improve with depth and model scale
• IC-1409FF blocks in BERT and GPT-2 modify token-to-token contextualization, with the effect concentrated in specific layers and targeting specific linguistic compositions rather than simple word co-occurrence
• IC-148Language models represent semantically equivalent inputs from different data types (languages, code, images, audio) close together in intermediate layers, with the shared space scaffolded by the model's dominant language
• IC-150Second-order effects of CLIP's MLP neurons are concentrated in late layers (8–10 of 12 in ViT-B/32)
• IC-1540CLS-token attention maps in pretrained ViT-t/16 exhibit high inter-layer correlation (cosine similarity up to 0.97) concentrated in layers 3–10, and MSA block outputs show high CKA in layers 2–8
• IC-1554LRE faithfulness in GPT-J is concentrated in intermediate layers and drops sharply in later layers, consistent with a mode switch from relational encoding to next-token prediction
• IC-1586In LLaMA-7B, steering all attention heads degrades JSON format accuracy below zero-shot, while steering a subset of 50-100 heads selected via multi-task profiling raises it to 96.64; performance varies dramatically across the 32 layers and individual heads
• IC-166A 1-dimensional subspace in a single layer encodes the context-versus-prior decision in Llama-3.1-8B, Gemma-2 9B, and Mistral-v0.3 7B, and setting this subspace steers the released (non-fine-tuned) models' behavior
• IC-181Truthfulness in Mistral-7B, Mistral-7B-Instruct, Llama3-8B, and Llama3-8B-Instruct is linearly decodable from internal representations at exact answer tokens, with middle-to-late layers being most informative
• IC-195Transformers achieve high brain alignment in early visual cortex at much shallower network depth than CNNs
• IC-200Personality-related neurons in Llama-3-8B-Instruct are concentrated in the deeper layers of the network
• IC-205Gemma 2's SAE features exhibit depth-dependent organization, with polysemantic features in early layers and persistent, matchable features in later layers
• IC-213Qwen2-0.5B-Instruct uses English-specific past tense heads and late FFN layers for morphological marking that is absent in Chinese
• IC-216DINOv2's value (V) facet from self-attention at layer n-1 encodes the most effective local features for VPR re-ranking, outperforming query and key facets, and layer n-1 outperforms the final layer n
• IC-218LLaMA3-8B and other LLMs solve arithmetic via a bag of independent heuristic neurons in middle and late MLP layers rather than a robust algorithm
• IC-249The GitHub data-refined LLC identifies the induction circuit heads in Pythia-70m by distinguishing previous-token and induction heads from other head types across layers 2 and 3
• IC-291ViT-B/16 and MAE-B/16 exhibit nearly inverted distributions of knowledge neurons across layers despite identical architecture and training data
• IC-298Weight similarity in open-source LLMs is organized in a depth-dependent structure with adjacent-layer similarity and distinct clusters at specific depths
• IC-301Yi-1.5-9B-chat exhibits a layer-repetition pattern where a section of layers is duplicated at a later depth
• IC-366In Llama-3-70B, activation sparsifiability varies systematically across depth: Wq/Wk peak in block 0 then decline sharply, Wo peaks at 80-90% mid-model, and Wdown is consistently more sparsifiable than Wgate and Wup
• IC-372Language models universally decompose retrieval tasks into request processing in middle layers and entity retrieval in late layers at the last token position
• IC-394Text generation in SDXL, DeepFloyd IF, and SD3 is controlled by less than 1% of parameters concentrated in specific cross- or joint-attention layers, and these layers are specialised for text content rather than visual template
• IC-402In LLaMA3-8B, LLaMA2-13B, and Mistral-7B, soft-prompt information flow peaks in shallow layers (2–10) and reasoning correctness depends on whether deeper layers redirect attention away from soft prompts to earlier reasoning steps
• IC-420CLIP ViT-B/16's SAE latent interpretability is depth-dependent: layer 11 encodes semantic object concepts while layers 2, 5, and 8 encode local shapes and attention patterns
• IC-426MultiBERTs exhibit a pushdown phenomenon where syntactic information migrates from later to earlier layers as training progresses
• IC-446In Stable Diffusion v1.5, concept-generating neurons are localized in the second layer of FFNs, spanning less than 3% of FFN parameters, and are disentangled from object-generating neurons
• IC-454Most hidden trajectories in trained LLMs exhibit exponential growth in norm as a function of depth, a property that emerges with training
• IC-462GPT-2 encodes toxicity in a low-dimensional linear subspace of its MLP layers, concentrated in higher layers
• IC-463DPO's first-step gradients in GPT-2 are correlated with the toxic subspace, with stronger alignment in later layers and with more samples
• IC-468Llama-3.1-405B's last hidden layer embeddings of erroneous tokens contain a linearly detectable error signal that a simple logistic regression head can exploit to flag incorrect continuations
• IC-482In Llama-3.1-8B, in-context graph structure is absent in early layers dominated by semantic priors and emerges clearly in deeper layers
• IC-501Linear probes on middle-layer attention heads of Llama-2-7B-Chat, Mistral-7B-Instruct-v0.1, and Vicuna-7B-v1.5 predict US lawmakers' DW-Nominate ideology scores with Spearman correlations around 0.85
• IC-503Adding probe regression coefficients to attention head activations steers Llama-2-7B-Chat, Mistral-7B-Instruct-v0.1, and Vicuna-7B-v1.5 toward more liberal or conservative generated text
• IC-547Llama3.2 3B and Llama3.1 8B exhibit saturation events in which the top prediction, once it appears at a given layer, remains unchanged through all subsequent layers
• IC-548CLIP-B/32 exhibits progressively increasing layer-wise representation similarity in both its vision encoder and text encoder, and the pattern also holds across modalities
• IC-561A linear direction in the residual stream at layer 16 of Llama3-8B-Instruct is causally necessary and sufficient for self-authorship claims: steering with it achieves 100% control over authorship assertions, and projecting it out reduces claims by 50-60%.
• IC-563The self-recognition vector's activation in Llama3-8B-Instruct is organized across depth: early layers (4-6) show diffuse perceptual activation to self-written text (present in both chat and base models), while layers 14-16 show a sharp decision-related peak at the output token that is present only in the chat model with role tags.
• IC-564The implicit attention matrices of Mamba, RWKV, and Griffin exhibit depth-dependent structure, with dependencies between distant tokens becoming more apparent in deeper layers
• IC-583Brain alignment in InstructBLIP and Idefics is organized by depth: middle layers align with higher visual regions while later layers align with early visual regions, whereas mPLUG-Owl shows later layers aligning with both
• IC-594In LLaMA-2-7B, the truth-correlated attention heads are concentrated after layer 9, with two functional types (structural and associative) evenly distributed throughout the upper portions of the model, showing no further depth-dependent specialisation within that region.
• IC-609The last transformer layer of Mistral 7B, Llama 3.2 1B, and Llama 3.2 3B shows anomalous trajectory statistics inconsistent with the linear drift-plus-noise pattern of intermediate layers
• IC-610In LLaMA2-7B-Chat, RAG hallucinations are causally driven by copying heads losing external context information during generation and by knowledge FFNs in mid-to-upper layers over-adding parametric knowledge to the residual stream
• IC-636Syntactic phenomena (determiner-noun and subject-verb agreement) localize to the same topmost-layer MLP neurons as factual information in BERT, GPT-2, and Llama-2
• IC-639The causal tracing pattern of MLP at early layers and attention at late layers is not stable across factual and syntactic phenomena in GPT-2 XL
• IC-646DINOv2, DeiT-III, and OpenCLIP repurpose approximately 2% of patch tokens in low-informative background areas as internal registers, discarding local patch information while aggregating global image information; DINO does not exhibit this behaviour
• IC-667CLIP's intermediate layer features encode object boundaries recoverable by k-means clustering, a property absent in shallow and deep layers
• IC-677CLIP ViT's image representation is primarily constructed by the last 4 MSA layers, with MLPs and early MSA layers contributing negligibly
• IC-686Adversarial perturbations alter CLIP ViT-B/32's token representations most strongly starting around layer 10
• IC-709Swin Transformer parameter redundancy is depth-dependent, with higher layers retaining fewer parameters than lower layers under module-aware pruning
• IC-715Factual information deleted from GPT-J, LLaMA-2, and GPT-2-XL via ROME or MEMIT remains linearly recoverable from intermediate hidden states, with up to 89% extraction success at budget b=20
• IC-808Sparse autoencoder features in Pythia-70m's residual stream are more interpretable than PCA, ICA, random, and default-basis directions, with the advantage declining from early to late layers
• IC-913In OPT-2.7B, Pythia-70M/1.4B/6.9B, and BERT-base, the stable rank of MLP lower layers shows a drop-and-bounce pattern during training that is more salient in top layers while bottom layers show suppressed dropping curves
• IC-917ViT-S models show early-layer sensitivity to layer-wise averaging, with the averaging direction being far more disruptive than random perturbations of the same norm
• IC-935In pre-trained ViT, query vector Kruskal rank reaches the context size only after one self-attention layer, while general position fails at all depths
• IC-936GPT-2's learned positional encodings cause context vectors to lose linear independence after one layer, whereas BERT's sinusoidal encodings preserve it
• IC-982In Stable Diffusion's UNET, visual attribute knowledge is distributed across multiple components with attribute-specific patterns, concentrated more in the up-block, and cross-attention layers are not the primary causal states
• IC-983In Stable Diffusion's CLIP text-encoder, knowledge about all visual attributes is localized to a single causal state: the first self-attention layer at the last subject token
• TM-011Spatial coherence of latent feature planes increases with encoder depth
• TM-012Longitude and time-of-year planes become circular in deeper encoder blocks