Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Does Progress On Object Recognition Benchmarks Improve Generalization on Crowdsourced, Global Data?
2024-01-16
· ICLR 2024 poster ·
anchor
Findings
IC-1469
Progress on standard ImageNet generalization benchmarks is 2.5x faster than progress on crowdsourced global data (DollarStreet, GEODE) across 98 vision models
IC-1470
Geographic disparities (Europe-Africa accuracy gap) are large across all 98 models and have more than tripled between least and best performing models on DollarStreet
IC-1471
Common robustness interventions (AugMix, CutMix, Deep AugMix, texture debiasing, antialiasing) and scaling of data or model size do not resolve geographic disparities in released vision models
IC-1472
DINOv2 (86M parameters) achieves the smallest GEODE geographic disparity (2.46% Europe-Africa gap) among all 98 models in the testbed