SourceCorrelating instruction-tuning (in multimodal models) with vision-language processing (in the brain)
Using variance partitioning on InstructBLIP representations, the paper measures the unique and shared brain variance explained by pairs of the 10 instructions. IC shows the highest shared variance with VQ1 (0.446), VQ2 (0.399), and CR (0.417), while showing lower shared variance with iu2 (0.341) and sr (0.369). In ROI-level analysis, IC retains unique variance in most high-level regions, and shared variance increases from early to higher visual areas, reflecting the hierarchical nature of visual processing. The authors note that the high overlap suggests MLLMs could improve in differentiating between instruction types.