The paper identifies individual neurons in Llama-2-7B whose input or output weights have high cosine similarity with the learned probe directions. These neurons are sensitive to the true spatial or temporal coordinates of entities across all entity types. Zero-ablating a space neuron (e.g., l20.7573) increases the model's next-token loss most in contexts involving geographic or climatic facts (up to 2.107 loss increase), and ablating a time neuron (e.g., l18.9387) increases loss most in contexts involving release dates (up to 2.254). Pinning a time neuron's activation to different values changes the model's predicted decade of publication for songs, movies, and books.
Evidence
interventional
Key metric
Time neuron l19.3610 has Spearman correlation 0.77 with art/entertainment release date; max loss increase 2.254 (time neuron l18.9387) and 2.107 (space neuron l20.7573) upon zero-ablation
Caveat
Experiments conducted only on Llama-2-7B; only two space neurons and two time neurons were ablated, and the loss-increase analysis is limited to a subset of Wikipedia articles.