SugarCrepe

anchor

Findings

• IC-1407OpenFlamingo and Idefics models perform near random chance on compositional image-text matching, and ICL has almost no effect on atomic foils [eval]
• IC-788CLIP ViT-B/32 fails to retrieve the correct image even when the generated target caption is well-aligned with the ground-truth image [context]