SourceFairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSs
On the FairFace dataset, using ten text prompts unrelated to facial or sensitive attributes (e.g., 'a photo of a criminal person'), zero-shot CLIP shows high maxSkew@1000 for both sex and race. CLIP ViT-B/32 achieves maxSkew of 0.206 (sex) and 0.743 (race), while CLIP ViT-L/14 achieves 0.206 (sex) and 0.768 (race). This indicates that CLIP's representations encode race and sex information that leaks into predictions for unrelated target attributes.