IC-636Syntactic phenomena (determiner-noun and subject-verb agreement) localize to the same topmost-layer MLP neurons as factual information in BERT, GPT-2, and Llama-2
Jingcheng Niu, Andrew Liu, Zining Zhu, Gerald Penn
Using integral-of-gradient attribution and neuron suppression, the paper shows that grammatical number in determiners is captured by just two neurons in BERT (w(10) 2096 for singular, w(9) 1094 for plural), with the same localisation metrics (|KN|, τ, r²₁) as ParaRel factual relations. The identified KNs occupy the topmost layers for both syntactic and factual phenomena, with no depth-based separation. Suppressing the plural neuron produces significant probability changes across all plural modifiers, but also affects words like 'scattered' that do not specify number, indicating reliance on co-occurrence frequency.
Evidence
interventional
Key metric
|KN| 1.86 (DNA.2) to 9.79 (DNA w. adj irr. 1) for BLIMP vs 0.167 (P101) to 28.993 (P20) for ParaRel on BERT; w(10) 2096 in 93% of 'this' pairs and 75% of 'that' pairs; w(9) 1094 in 100% of 'these' and 'those' pairs
Caveat
The patterns identified by the neurons deviate from strict grammatical rules, affecting modifiers that do not specify number (e.g., 'scattered', 'any', 'all'), suggesting the neurons capture co-occurrence statistics rather than pure syntactic features.