SourceLanguage Models Represent Space and Time
The paper identifies individual neurons in Llama-2-7B whose input or output weights have high cosine similarity with the learned probe directions. These neurons are sensitive to the true spatial or temporal coordinates of entities across all entity types. Zero-ablating a space neuron (e.g., l20.7573) increases the model's next-token loss most in contexts involving geographic or climatic facts (up to 2.107 loss increase), and ablating a time neuron (e.g., l18.9387) increases loss most in contexts involving release dates (up to 2.254). Pinning a time neuron's activation to different values changes the model's predicted decade of publication for songs, movies, and books.