Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
2024-01-16
· ICLR 2024 spotlight ·
anchor
Findings
IC-925
MultiBERTs exhibits the same phase transition pattern (UAS spike, loss drop, BLIMP improvement) as the authors' own BERT-base training
IC-926
Across 25 MultiBERTs seeds, UAS does not correlate with MLM test loss or BLIMP accuracy