Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Layer-wise linear mode connectivity
2024-01-16
· ICLR 2024 poster ·
anchor
Findings
IC-916
Pythia models show scale-dependent last-layer averaging barriers: 70M exhibits a barrier of ~13 while 410M shows ~1
IC-917
ViT-S models show early-layer sensitivity to layer-wise averaging, with the averaging direction being far more disruptive than random perturbations of the same norm