Register tokens

A transformer reuses a few patch tokens as internal scratch space unrelated to the content of those patches. They surface in attention and attribution maps as fixed positions that light up whatever the input shows.

Findings

• IC-526GPT-2 small's first token position has residual stream norms more than an order of magnitude larger than all other positions
• IC-646DINOv2, DeiT-III, and OpenCLIP repurpose approximately 2% of patch tokens in low-informative background areas as internal registers, discarding local patch information while aggregating global image information; DINO does not exhibit this behaviour
• TM-010Fixed corner patches dominate Integrated Gradients maps regardless of image content