IC-521MLLMs show asymmetric modality-specific perception, with Gemini Pro achieving 69.97% on visual-only questions but only 24.45% on audio-only, while Video-Chat outperforms ChatUniVi on audio despite worse visual scores
Xuehai He, Weixi Feng, Kaizhi Zheng, Yujie Lu, Wanrong Zhu, Jiachen Li, Yue Fan, Jianfeng Wang, Linjie Li, Zhengyuan Yang, Kevin Lin, William Yang Wang, Lijuan Wang, Xin Eric Wang
Using the synthetic subsets of MMWorld, the paper isolates audio and visual perception by generating QA pairs from a single modality. Gemini Pro performs best on visual-only questions (69.97% average) but drops to 24.45% on audio-only. Video-Chat shows better audio perception (38.82%) than ChatUniVi (31.82%) despite worse visual perception (39.07% vs 48.44%), which the authors attribute to Video-Chat's use of the Whisper speech recognition model. Video-LLaMA and Otter perform near or below random on both modalities.
Evidence
correlational
Key metric
Gemini Pro: 69.97% visual vs 24.45% audio; Video-Chat: 38.82% audio vs 39.07% visual; ChatUniVi: 31.82% audio vs 48.44% visual; random: 32.44% audio, 30.91% visual
Caveat
Only 5 models were evaluated on the synthetic subsets (Video-Chat, ChatUniVi, Video-LLaMA, Otter, Gemini Pro).