IC-679CLIP relies on background/location as a spurious cue for bird classification, and ablating geolocation heads improves worst-group accuracy by 25.2%
Yossi Gandelsman, Alexei A Efros, Jacob Steinhardt
On the Waterbirds dataset, where image background (water/land) is a spurious correlate of bird type (waterbird/landbird), the paper identifies 'geolocation' and 'image-location' heads via TextSpan and mean-ablates their direct contributions. This increases worst-group accuracy from 48% to 73% for ViT-L (25.2% gain), without any additional training. A baseline of ablating 10 random heads (best of 5 trials) does not achieve this improvement. Per-class results show the spurious correlation between background and class is specifically reduced.
Evidence
interventional
Key metric
Worst-group accuracy on Waterbirds: ViT-L 48%→73% (25.2% gain), ViT-B/16 45.6→57.5, ViT-H/14 37.2→43.3. Per-class (ViT-L): waterbird on land 66.2→77.8, landbird on water 47.7→72.9.
Caveat
Head selection is manual (authors annotated roles from TextSpan descriptions); only one spurious cue (geolocation) is tested; the improvement is on a single dataset.