IC-913In OPT-2.7B, Pythia-70M/1.4B/6.9B, and BERT-base, the stable rank of MLP lower layers shows a drop-and-bounce pattern during training that is more salient in top layers while bottom layers show suppressed dropping curves

Yuandong Tian, Yiping Wang, Zhenyu Zhang, Beidi Chen, Simon Shaolei Du

SourceJoMA: Demystifying Multilayer Transformers via Joint Dynamics of MLP and Attention

The paper examines public intermediate checkpoints of OPT-2.7B and Pythia-70M/1.4B/6.9B to verify its theoretical prediction that attention first becomes sparse then dense. While the attention entropy patterns show only a less salient drop-and-bounce, the stable rank of the MLP lower-layer projection matrix shows a much more pronounced drop-and-bounce structure in top layers. In bottom layers, the stable rank shows only dropping curves, suppressed by top-level learning. The same pattern is confirmed in BERT-base (encoder-decoder), where attention entropy and stable rank behave similarly to the decoder-only case.

Evidence

correlational

Caveat

The attention entropy patterns in the pre-trained models are described as 'less salient' compared to the from-scratch experiments; the stable rank signal is the more reliable indicator per the authors.

Model

OPT, Pythia, BERT

Concepts

Depth-dependent structure

Datasets

Wikitext-103 [eval]

Related findings

IC-914, IC-915

Extraction

automatic-extraction