SourceAnalyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps
The paper shows that the contextualization change introduced by FF is progressively undone by the subsequent residual connection and layer normalization. The L2 norm of the res2 bypass vector is more than twice that of the FF output in most layers, so the original representation dominates. Additionally, the FF output has outlier values in specific dimensions, and the LN weight γ is negatively correlated with these dimensions (Pearson r from -0.45 to -0.78 in BERTs, -0.22 to -0.59 in GPT-2). When the bottom 1% of dimensions with the lowest γ are excluded from the norm calculation, the FF contextualization change drops from 0.21 to 0.09 in BERT and from 0.15 to 0.02 in GPT-2, indicating that FF's effect is realized through very specific dimensions that LN erases. This suggests redundancy in the transformer layer's processing.