SourceDifferentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient
The authors apply their weight- and data-refined local learning coefficient (wdrllc) to Pythia-70m, a 70M-parameter released transformer. Using the GitHub code distribution as the data refinement, the relative difference between Pile and GitHub LLC values for layer 2 heads reveals a single outlier corresponding to the model's sole previous-token head. For layer 3, the same measure separates two clusters: candidate induction heads and non-induction heads. These identifications are confirmed using the previous-token score and prefix-matching score from Olsson et al. (2022), with head 3:2 noted as an exception to the induction-head cluster.