Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
How to Fine-Tune Vision Models with SGD
2024-01-16
· ICLR 2024 poster ·
anchor
Findings
IC-1099
AdamW-pretrained vision models (ViTs, ConvNeXt) have disproportionately large embedding-layer gradients at initialization, causing SGD fine-tuning to degrade OOD accuracy by up to 15% relative to AdamW