IC-644CLIP relies on spurious features (background) as a shortcut in zero-shot classification, and conditioning on the correct background reduces this reliance
Bang An, Sicheng Zhu, Michael-Andrei Panaitescu-Liess, Chaithanya Kumar Mummadi, Furong Huang
On the Waterbirds dataset, CLIP's Grad-CAM saliency shows 37.2% of attention on spurious (background) regions when no context is provided. Conditioning on the correct background shifts this to 33.7% spurious and 66.3% core (ViT-B/32). This shortcut manifests as a large group-robustness gap: on Waterbirds with ViT-L/14, the gap between average and worst-group accuracy is 45.91 without context, reducing to 41.51 with background. On CelebA (hair color), the gap is 9.61 without context, reducing to 5.99 with gender. The paper attributes the bias to CLIP associating water backgrounds with waterbirds rather than focusing on the bird itself.
Evidence
correlational
Key metric
Grad-CAM core saliency 62.8% to 66.3%, spurious 37.2% to 33.7% (ViT-B/32, Waterbirds); Waterbirds ViT-L/14 gap 45.91 to 41.51 (with background); CelebA ViT-L/14 gap 9.61 to 5.99 (with gender)
Caveat
The group robustness experiments are conducted on the training set of Waterbirds and CelebA. The Grad-CAM analysis uses ViT-B/32 while the group robustness tables use multiple variants; the effect is not uniform across all variants (e.g., ViT-B/16 on Waterbirds shows a gap increase from 60.56 to 66.91 with background).