IC-290Zeroing out or doubling specific FFN neurons identified by the neuron path method causes significant accuracy changes in ViT and MAE models

Yifan Wang, Yifei Liu, Yingdong Shi, Changming Li, Anqi Pang, Sibei Yang, Jingyi Yu, Kan Ren

SourceDiscovering Influential Neuron Path in Vision Transformers

The paper identifies one influential neuron per FFN layer (a 'neuron path') using a joint attribution score, then intervenes by zeroing out or doubling those neuron values. Across all four models, the neuron path method produces the largest accuracy deviations: removing the path neurons drops accuracy by 2.40% (ViT-B/16), 3.81% (ViT-B/32), 5.28% (ViT-L/32), and 26.50% (MAE-B/16), while doubling them improves accuracy by 2.04%, 3.06%, 5.02%, and 7.28% respectively. Baseline methods (max activation, influence pattern) produce much smaller effects, confirming that the identified paths carry critical information for inference.

Evidence
interventional
Key metric
Removal accuracy deviation: -2.40% (ViT-B/16), -3.81% (ViT-B/32), -5.28% (ViT-L/32), -26.50% (MAE-B/16); Enhancement accuracy deviation: +2.04%, +3.06%, +5.02%, +7.28% respectively. JAS: 0.4078, 0.6610, 1.0086, 0.0095.
Caveat
The method only targets FFN neurons (first linear layer output), not attention or other components. The paper acknowledges this as a limitation.
Model
ViT ViT-B/16, ViT-B/32, ViT-L/32, MAE-B/16
Concepts
Explanation faithfulness
Datasets
ImageNet-1k / ImageNet / ImageNet-1k-val / ImageNet-Val [eval]
Methods
Knowledge Attribution [compared-to]
Related work
Knowledge Neurons in Pretrained Transformers [builds-on]
Related findings
IC-291, IC-292, IC-293
Extraction
automatic-extraction