On VQA-X, LMMs are given an image, question, and correct answer, then asked to generate an explanation. Zero-shot CIDEr scores are significantly below SOTA: OFv2-9b scores 56.17 and Idefics-9b scores 67.43. ICL with more demonstrations substantially improves CIDEr: OFv2-9b rises to 80.51 at 32-shot, Idefics-9b to 90.18. There is a clear positive correlation between model size and explanation quality. Idefics-80b surpasses SOTA with ICL, but instruction tuning does not help the 80B model, possibly due to overfitting when training the LLM.
The paper notes that for 80B-parameter models, instruction tuning does not improve explanation quality, which might be due to overfitting when training the LLM. CIDEr measures syntactic similarity to human annotations, not semantic faithfulness of the explanation.