IC-192CLIP's contrastive image-text training objective hinders its ability to rank or order images, yielding near-chance performance on ranking tasks in both zero-shot and fine-tuned settings
The paper measures CLIP (ConvNeXt-L, LAION-5B) on four ranking tasks: facial age estimation, historical image dating, object count sorting, and image quality/aesthetics assessment. In zero-shot mode, CLIP achieves only 43.3% accuracy (MAE 0.80) on ADIENCE age estimation, 26.1% accuracy (MAE 1.48) on HCI dating, and PLCC 0.026 / SRCC 0.001 on object count sorting. Even after fine-tuning with standard contrastive objectives, CLIP reaches only PLCC 0.251 / SRCC 0.422 on object counting and PLCC 0.245 / SRCC 0.216 on KONIQ-10k. The paper attributes this failure to CLIP's training objective, which matches image content to sentences and captures global information but does not learn cross-image relational structure needed for ordering.
The paper does not ablate the training objective in isolation to confirm causality; the attribution to the contrastive objective is stated as motivation rather than tested experimentally. The fine-tuned CLIP baseline uses the same contrastive objective, so it is unclear whether a different fine-tuning objective would close the gap.