The paper tracks the degree of emergence (a structural metric combining heterogeneity and regularity changes) alongside standard benchmark scores for Pythia 160M, 1.4B, and 2.8B across training epochs. The structural metric rises sharply in the first ~15,000 epochs, then gradually increases to ~40,000 epochs, then stabilizes with fluctuations. This trajectory mirrors the improvement in LAMBADA, PIQA, ARC-Easy, and SCIQ scores. The authors note significant performance oscillations on WSC and LogiQA, suggesting limited utility of those benchmarks.
The authors acknowledge the correlation is restricted to the specific datasets tested and that a more comprehensive evaluation across more datasets is needed. The relationship is correlational, not causal.