IC-1127The LM head in GPT-2, GPT-J, BLOOM, Pythia, and LLaMA-2 projects all input token hidden states into interpretable token distributions over the vocabulary, and these distributions converge approximately monotonically toward the final layer's distribution

Zijian Feng, Hanzhang Zhou, ZIXIAO ZHU, Junlang Qian, Kezhi Mao

SourceUnveiling and Manipulating Prompt Influence in Large Language Models

Prior work on the logit lens showed that the last token's hidden state can be projected through the LM head to yield an interpretable next-token distribution. This paper extends that observation to all input tokens at every layer, validating it empirically on five model families. Qualitative visualizations show that intermediate-layer projections produce meaningful, context-appropriate predictions (e.g., 'some boys discover' yields 'themselves' in mid-layers of LLaMA-2). Quantitatively, the mean KL divergence between each layer's projected distribution and the final layer's distribution decreases in an approximately monotonic fashion, indicating that each token's contribution converges synchronously at the final layer. This property holds across GPT-2, GPT-J, BLOOM, Pythia, and LLaMA-2 on diverse linguistic phenomena from BLIMP.

Evidence

correlational

Caveat

Convergence is described as 'approximately monotonic' rather than strictly monotonic; primary evidence is qualitative visualizations supplemented by mean KL divergence across layers, with no specific divergence values reported in the text.

Model

GPT-2, GPT-J, BLOOM, Pythia, Llama 2 / Llama 2 base

Concepts

Linear representation, Depth-dependent structure

Datasets

BLIMP [eval]

Related work

Interpreting GPT: The Logit Lens [builds-on], Transformer Feed-Forward Layers are Key-Value Memories [builds-on], Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space [builds-on], Analyzing Transformers in Embedding Space [builds-on]

Extraction

automatic-extraction