SourceBoosting the visual interpretability of CLIP via adversarial fine-tuning
The paper measures the stability of Grad-CAM saliency maps by injecting small random Gaussian noise into the input and computing the SSIM between the noisy and clean Grad-CAM maps. For the original CLIP ViT-B/16, even a tiny noise level of 1/255 reduces SSIM to 91.18%, and at 9/255 it drops to 70.58%. In contrast, the AFT-fine-tuned CLIP maintains SSIM above 98.55% across all noise levels. The authors attribute the original CLIP's fragility to the non-smoothness in its decision boundary introduced by false negatives during contrastive training.