IC-251CLIP's overall MMEB performance drops by 29.4% when task-specific instructions are prepended to queries, with classification degrading by 59.3%

Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, Wenhu Chen

SourceVLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks

The paper measures CLIP's performance on all 36 MMEB datasets both with and without task-specific instructions prepended to the query. Without instructions, CLIP achieves 37.8 overall precision@1. With instructions, it drops to 26.7, a 29.4% relative decrease. The largest degradation is in classification (-59.3%, from 42.8 to 17.4), while visual grounding is nearly unaffected (+2.1%). This contrasts sharply with the authors' VLM2Vec model, which improves 49.4% with instructions.

Evidence

correlational

Key metric

CLIP w/o instruction: 37.8 overall; w/ instruction: 26.7 overall; Δ -29.4% overall, -59.3% classification, -12.1% VQA, -22.1% retrieval, +2.1% grounding (precision@1, 36 MMEB datasets)

Caveat

The paper does not specify which CLIP variant is used in this ablation; the instruction format is the same task-specific instructions used for VLM2Vec training.

Model

CLIP / CLIP-ViT (LC)

Concepts

Failure mode

Related findings

IC-250

Extraction

automatic-extraction