IC-1005Social bias neurons in BERT-base-cased and RoBERTa-base are concentrated in the deepest transformer layers

Yan Liu, Yu Liu, Xiaokang Chen, Pin-Yu Chen, Daoguang Zan, Min-Yen Kan, Tsung-Yi Ho

SourceThe Devil is in the Neurons: Interpreting and Mitigating Social Biases in Language Models

Using the IG² attribution method, the paper identifies specific feed-forward neurons in BERT-base-cased and RoBERTa-base that are responsible for the logits gap between demographic pairs. These social bias neurons are not uniformly distributed across layers: they are predominantly concentrated in the deepest three layers (10th, 11th, 12th), with the single deepest layer (12th) holding the largest share. This depth-dependent organisation means the bias is most strongly expressed near the output layer.

Evidence

correlational

Key metric

highest concentration in layer 12, accounting for approximately 55.7% (BERT) and 36.4% (RoBERTa) of the total quantity; neurons concentrated in layers 10-12

Caveat

The attribution is limited to feed-forward neurons at the masked token position; other model components (attention, other positions) are not examined.

Model

BERT, RoBERTa / RoBERTa-L

Concepts

Depth-dependent structure

Datasets

StereoSet [eval]

Related work

Integrated Gradients / Integral of gradients [builds-on]

Extraction

automatic-extraction