SourceNot All Language Model Features Are One-Dimensionally Linear
The authors construct a synthetic dataset of prompts like '[very early/very late] on [monday/tuesday/.../sunday]' and project the layer-30 activations into the PCA plane of the weekday representations. The 'very early' version of each weekday maps towards the previous weekday and the 'very late' version maps towards the next weekday along the circle. A similar result is shown for '[morning/evening] on [weekday]' prompts. This demonstrates that the circular feature encodes a continuous notion of time rather than just discrete weekday labels.