SourceTracing Representation Progression: Analyzing and Enhancing Layer-Wise Similarity
The authors apply the logit lens (last-layer classifier) to intermediate hidden states of Llama3.2 3B (24 layers) and Llama3.1 8B (32 layers) using the same prompt and greedy decoding. They observe that the model's predicted token stabilises at a certain layer and does not change in any deeper layer, a phenomenon previously described by Geva et al. (2022) for smaller models. A specific instance in Llama3.2 3B with the prompt 'simply put, the theory of relativity states' is visualised across 24 decoding steps, showing the predicted word locking in at an intermediate layer. This confirms that the saturation-event phenomenon generalises to large, recently released LLMs.