SourceRephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models
The paper compares three selection strategies: using the original question, selecting among PEGASUS-generated paraphrases via model confidence, and selecting among REPARe-generated semantically enriched questions via model confidence. For BLIP-2, paraphrase selection yields 40.23% on A-OKVQA direct (vs. 41.86% baseline, −1.63) and 73.57% on A-OKVQA MC (vs. 73.89%, −0.32), while REPARe selection reaches 44.87% and 77.20% respectively. MiniGPT-4 shows the same pattern: paraphrase selection gives 26.28% (A-OKVQA direct, vs. 27.51% baseline) and 40.52% (MC, vs. 41.66%), while REPARe selection reaches 33.23% and 63.20%. This indicates the models' self-assessed confidence is unreliable for choosing among cosmetic rewrites but effective for semantically enriched candidates.