IC-156Gemini models show substantially lower confusion scores than other MLLMs yet still perform at or near random guessing, suggesting their bottleneck is medical knowledge or reasoning rather than visual encoding
Mohammad Shahab Sepehri, Zalan Fabian, Maryam Soltanolkotabi, Mahdi Soltanolkotabi
Gemini 1.5 Pro and Gemini 2.0 Flash have confusion scores of 58.52% and 67.05% respectively, far below the 80-100% seen in most other models. This means they do differentiate between the two images in a pair more often than other models. However, their set accuracies (19.89% and 29.55%) are still at or only slightly above the 25% random baseline. The authors hypothesize that Gemini's visual representations are rich enough to distinguish the images, but the models lack the medical knowledge or reasoning skills to map that distinction to the correct answer. Gemini 2.0 Flash achieves the best individual accuracy (61.93%) and the best category-specific result (42.86% set accuracy on nuclear medicine).