SourceSudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
The authors examine the 5 MultiBERTs runs that have intermediate checkpoints available and observe the same qualitative pattern they found in their own BERT-base runs: UAS shows a sharp initial spike followed by a plateau within 20k timesteps, the BLIMP increase also occurs within the first 20k steps, and the loss drops precipitously initially before declining more slowly. The timing of the UAS plateau implies a slightly faster timeline for linguistic structure acquisition compared to the authors' reproduction, and the loss is slightly lower with a slightly higher BLIMP average. The authors conclude that MultiBERTs is compatible with the same phase transition.