SourceDetecting Backdoor Samples in Contrastive Language Image Pretraining
The authors identified 798 images of a birthday cake with the caption 'the birthday cake with candles in the form of number icon' in the CC3M dataset, accounting for roughly 0.03% of the data. Using an adapted trigger recovery method based on Neural Cleanse, they distilled a trigger pattern from the OpenCLIP released model (ResNet-50, trained on CC12M). When the recovered trigger is attached to ImageNet test images in zero-shot classification, the model produces the target output with 98.92% attack success rate, confirming that the unintentional backdoor has been learned into this popular open-source model. A counter-example with random initialization of the trigger pattern yields only 45.37% and 13.09% ASR, supporting that the birthday cake pattern is a genuine learned shortcut rather than a targeted adversarial perturbation.