By probing multiple syntactic properties (NER, POS, dependency, depth, distance, phrase boundaries) across all layers of MultiBERT checkpoints at many training steps, the paper finds that early in training, syntactic information is best recovered from later, more contextualized layers. As training proceeds, the same properties become better encoded in earlier layers, until only the first couple of layers are needed. The rate of pushdown varies by property, following an order: NER, phrase start, c-POS/f-POS, phrase end, dependency, depth, distance.
Evidence
observational
Caveat
The pushdown is measured via linear probes on templatic and treebank data; the paper notes the timing may reflect rule complexity or a dependency hierarchy but does not disentangle these.