Circular representation

A quantity that wraps around, such as longitude or time of year, is encoded in a closed loop rather than along a straight direction. It is the clearest counterexample to reading every encoded property as a linear direction, and it cannot be recovered by a linear read-out.

Findings

• IC-104Value anchor prompting produces LLM value correlation structures that closely match the human circular value structure, while standard prompting does not
• IC-105Value anchoring produces a sinusoidal scoring pattern around the circular value structure, with scores decreasing as circular distance from the anchor increases
• IC-358GPT-2-small and Mistral 7B contain circular representations of days of the week and months of the year in their internal activations, discovered via SAE dictionary element clustering
• IC-359Mistral 7B and Llama 3 8B causally use circular subspaces to compute modular arithmetic on days of the week and months of the year
• IC-360GPT-2 achieves only trivial accuracy on modular arithmetic tasks for days of the week and months of the year despite containing circular representations
• IC-361Mistral 7B's circular representation of days of the week is continuous, mapping intermediate time-of-day values to positions between adjacent weekdays
• TM-012Longitude and time-of-year planes become circular in deeper encoder blocks