IC-1409FF blocks in BERT and GPT-2 modify token-to-token contextualization, with the effect concentrated in specific layers and targeting specific linguistic compositions rather than simple word co-occurrence
Using a norm-based attention map analysis extended with integrated gradients to decompose the nonlinear FF activation, the paper measures how much the FF block changes the token-to-token attention map relative to the attention block alone. In BERT-base the by-layer average contextualization change by FF is 0.21, concentrated in the 3rd and 9th-11th layers; in GPT-2 it is 0.15, concentrated in early layers. The amplified token pairs are not explained by co-occurrence (Spearman correlation with PMI is only 0.06-0.14) but instead cluster into linguistic categories such as subword-to-word and compound-noun constructions. Results are consistent across 11 BERT/RoBERTa variants, GPT-2, and OPT-125M on both Wikipedia and SST-2.
Evidence
correlational
Key metric
by-layer average contextualization change (1-ρ) by FF: 0.21 (BERT-base) and 0.15 (GPT-2); by LN: 0.27 (BERT) and 0.38 (GPT-2); Spearman correlation between ff-amp scores and PMI: 0.06-0.14 across layers
Caveat
The linguistic category labels were assigned by one of the authors manually; the analysis is at the subword level and the specific layer locations differ between BERT (middle-to-late) and GPT-2 (early), with the cause of this gap left as future work.