SourceSuccessor Heads: Recurring, Interpretable Attention Heads In The Wild
The effective OV circuit of the successor head shows that logits on or above the main diagonal (output token ≤ input token) are lower than logits below the diagonal (output token > input token), for cardinal words, numbers, and other ordinal tasks. This means the head is biased toward predicting values greater than the input. As a consequence, vector arithmetic using mod-10 features to shift toward a lower ordinal value fails: for example, wu·Wov(MLP0(We('35')) − kf5 + kf3) places high logits on '43' rather than '33', a +10 effect occurring for 2/3 of entries below the diagonal in the 20–29 numbers table. The mod-10 features themselves do not exhibit this bias, indicating the head uses additional numeric information beyond the mod-10 features.