IC-236Zero-shot Grounding-DINO and Florence-2 show a significant performance gap on referring expression comprehension compared to their fine-tuned versions
Amaia Cardiel, Eloi Zablocki, Elias Ramzi, Oriane Siméoni, Matthieu Cord
The paper measures the zero-shot p@1 of Grounding-DINO (Swin-T) and Florence-2 (Large) on four REC datasets and compares them to the same models after white-box fine-tuning on REC data. The gap ranges from -11.5 to -37.1 p@1 depending on the model and dataset. For example, Florence-2 drops from 91.02 p@1 (fine-tuned) to 66.16 p@1 (zero-shot) on RefCOCOg test, and Grounding-DINO drops from 77.99 to 59.32. The authors note that while these VLMs can detect most objects mentioned in a query, they struggle to select only the described object among distractors.
Evidence
correlational
Key metric
Zero-shot vs fine-tuned gap: -11.5 to -37.1 p@1. FLO2 RefCOCOg test: 66.16 (zero-shot) vs 91.02 (FT). GD RefCOCOg test: 59.32 (zero-shot) vs 77.99 (FT). FLO2 RefCOCO test: 57.21 (zero-shot) vs 93.42 (FT).
Caveat
The fine-tuned versions use different backbones (Swin-B for GD, full fine-tuning for FLO2), so the gap conflates zero-shot limitation with backbone size difference for GD.