IC-250Existing multimodal embedding models show highly uneven performance across MMEB's four meta-task categories, with VQA scores as low as 4.2 and overall scores ranging from 13.3 to 44.7
The paper evaluates seven released multimodal embedding models on the new MMEB benchmark, which spans classification, VQA, retrieval, and visual grounding. All baselines without fine-tuning show a pronounced gap between task categories: CLIP scores 53.0 on retrieval but only 9.1 on VQA; BLIP2 scores 47.0 on grounding but 4.2 on VQA; E5-V scores 11.5 on retrieval but 4.9 on VQA. The best baseline without fine-tuning (UniIR CLIP SF) reaches 44.7 overall, and even with fine-tuning the best (OpenCLIP) reaches only 47.2. No single baseline achieves reasonable performance across all four meta-task categories simultaneously.
The paper notes that for CLIP-family baselines, some queries or target text may be truncated due to text encoder length limitations, and score-level fusion with equal weights is used without instructions.