The paper visualizes attention maps across all layers and heads of LLaMA-2-7B and LLaMA-2-70B, finding that beyond the bottom two layers the model heavily attends to the initial token. A substitution experiment on LLaMA-2-13B shows that replacing the first four tokens with linebreak characters still restores perplexity to near-normal levels (5.60 vs 5.40), confirming the effect is positional rather than semantic. Across MPT-7B, Falcon-7B, Pythia-12B, and LLaMA-2-7B, introducing four initial tokens alongside a recent window restores perplexity from catastrophic levels to near-normal. BERT-base-uncased shows the same pattern with its [SEP] token receiving disproportionately high attention in most layers. The authors attribute the phenomenon to the softmax constraint requiring attention scores to sum to one, forcing the model to dump excess attention onto globally visible initial tokens.
The attention sink observation is directly visualized on LLaMA-2-7B and LLaMA-2-70B; other model families are supported by perplexity recovery experiments rather than direct attention map visualization. The BERT result is a single attention map on one sentence.