The paper computes layer-wise linear interpolation barriers for three released Pythia model sizes (70M, 160M, 410M) on WikiText. For most layers, no significant barrier appears, consistent with the paper's main claim of layer-wise linear mode connectivity. However, the last layer (embed_out) shows a pronounced barrier in the smallest model (70M, reaching ~13.4) that diminishes with scale (160M ~7.0, 410M ~1.0). This scale-dependent pattern means that layer-wise averaging of the final layer is problematic for small Pythia models but not for larger ones.
Evidence
correlational
Key metric
embed_out relative barrier: 70M ~13.4, 160M ~7.0, 410M ~1.0 (interpolated into model1, end of training)
Caveat
Pythia models were trained on different datasets; the barrier is measured on WikiText test set. The paper notes that different initialization and small learning rates can result in barriers in some shallow layers.