The paper evaluates 10 LMMs on COCO captioning using the CHAIRS metric to measure object hallucinations. All models show high zero-shot CHAIRS scores, with Idefics-9b reaching 31.42 and OFv1-9b at 17.38. While 4-shot ICL slightly reduces CHAIRS (e.g., OFv2-9b drops from 7.21 to 5.02), increasing to 8, 16, or 32 shots reverses the trend, with CHAIRS rising to 9.00 for OFv2-9b and 9.56 for Idefics-9b at 32 shots. The amplification is most severe for smaller models. Pretraining on more data (OFv2 vs OFv1) and instruction tuning reduce baseline hallucination.
The paper notes that Idefics models show very high hallucination when evaluated in zero-shot a la flamingo (2-shot without images) versus true zero-shot, suggesting the evaluation protocol affects results.