The authors examine the 5 MultiBERTs runs that have intermediate checkpoints available and observe the same qualitative pattern they found in their own BERT-base runs: UAS shows a sharp initial spike followed by a plateau within 20k timesteps, the BLIMP increase also occurs within the first 20k steps, and the loss drops precipitously initially before declining more slowly. The timing of the UAS plateau implies a slightly faster timeline for linguistic structure acquisition compared to the authors' reproduction, and the loss is slightly lower with a slightly higher BLIMP average. The authors conclude that MultiBERTs is compatible with the same phase transition.
Evidence
observational
Key metric
uas clearly shows a sharp initial spike followed by a plateau within 20k timesteps; the blimp increase also occurs within the first 20k steps; the loss drops precipitously initially and more slowly after the 20k step checkpoint
Caveat
The intermediate checkpoints are not granular enough to show the exact timing of the UAS spike or a clear break in the BLIMP accuracy curve; the confirmation is qualitative rather than quantitative.