SourceLayer-wise linear mode connectivity
The paper evaluates three pairs of pre-trained ViT-S models (differing in learning rate, data augmentation, or SAM radius) from a public repository. Layer-wise interpolation at alpha=0.5 is barrier-free for almost all layers, but the most sensitive layers are the early attention (to-qkv) and fully-connected (net) weights, with barriers reaching ~4-5 in early layers versus ~0 in deep layers. When comparing the loss increase from layer-wise interpolation to random perturbations of the same norm, the interpolation direction causes substantially higher loss, indicating that the averaging direction is a special, high-curvature direction on the loss surface. Flatter models (larger learning rate or larger SAM rho) show slower loss growth over the interpolation coefficient.