IC-837CLIP ViT-L/14 embeds more target-attribute information and less sensitive-attribute information than CLIP ResNet-50 on Waterbirds

Sepehr Dehdashtian, Lan Wang, Vishnu Boddeti

SourceFairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSs

Using HSIC and KCC to measure statistical dependence between CLIP features and ground-truth labels on Waterbirds, CLIP ViT-L/14 shows higher dependence with the target attribute (HSIC 0.1849 vs 0.1423) and lower dependence with the sensitive attribute (HSIC 0.2392 vs 0.3823) compared to CLIP ResNet-50. This explains why debiasing methods perform better on ViT-L/14: the larger model's features are more informative for the target while carrying less of the spurious sensitive signal.

Evidence

correlational

Key metric

HSIC(x,y): ViT-L/14 0.1849, ResNet-50 0.1423; HSIC(x,s): ViT-L/14 0.2392, ResNet-50 0.3823; KCC(x,y): 0.8267 vs 0.7556; KCC(x,s): 0.8661 vs 0.8861

Caveat

Measured only on Waterbirds; the paper notes this explains the performance discrepancy between the two CLIP variants in Table 2.

Model

CLIP / CLIP-ViT (LC)

Concepts

Scale-dependent behaviour

Datasets

Waterbirds [eval]

Methods

HSIC [primary]

Related findings

IC-834, IC-835, IC-836

Extraction

automatic-extraction