Norm-based Analysis

anchor

Findings

• IC-1409FF blocks in BERT and GPT-2 modify token-to-token contextualization, with the effect concentrated in specific layers and targeting specific linguistic compositions rather than simple word co-occurrence [primary]
• IC-1410FF's contextualization effects in BERT and GPT-2 are largely canceled by the residual connection and layer normalization, with LN's γ weights specifically shrinking the outlier dimensions in FF output [primary]