ConvNeXt

image · discriminative

Variants

ConvNeXt-B, ConvNeXt-L

Findings

• IC-1017Vision and language models pre-trained on noisy data exhibit degraded OOD transfer that is partially recoverable via SVD-based feature-space regularization
• IC-1099AdamW-pretrained vision models (ViTs, ConvNeXt) have disproportionately large embedding-layer gradients at initialization, causing SGD fine-tuning to degrade OOD accuracy by up to 15% relative to AdamW
• IC-1469Progress on standard ImageNet generalization benchmarks is 2.5x faster than progress on crowdsourced global data (DollarStreet, GEODE) across 98 vision models
• IC-1470Geographic disparities (Europe-Africa accuracy gap) are large across all 98 models and have more than tripled between least and best performing models on DollarStreet
• IC-295The ViT model's ECE can be reduced to near-zero by trivial mean-replacement recalibration while maintaining test accuracy, but NLL increases from 65.35 to 144.66, demonstrating that ECE and accuracy alone are an insufficient reporting standard for calibration
• IC-673Trained depthwise convolutional kernels in DS-CNN architectures converge to identifiable DoG-like patterns, with over 95% of ConvNeXtV2 and over 90% of ConvNeXt filters classifiable into a small set of clusters

Shared mechanisms

• Method artefactalso in Baichuan, CLIP / CLIP-ViT (LC), EfficientNet, Falcon, Gemma, Gemma 2, GPT-2, GPT-3.5 / ChatGPT-3.5, GPT-4o, GPT-J, InternLM, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, MAP-NEO, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, OLMo / OLMo base, OpenCLIP, OpenLLaMA, OPT, Pythia, Qwen1.5, Qwen 2, Qwen2.5, RedPajama, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, Skywork, Stable Diffusion, StableLM, TerraMind, ViT, Yi, ZiYA2