IC-163LLaVA-1.5, LLaVA-Next, and GPT-4V show near-zero accuracy on GUI grounding benchmarks while achieving 50-85 on general image grounding (RefCOCO+), indicating a failure mode specific to GUI grounding scenarios
The paper evaluates multiple released MLLMs on both general image grounding (RefCOCO+) and GUI-specific grounding (element grounding and action grounding on VisualWebBench, ScreenSpot, RefExp). LLaVA-1.5-7B scores 50.0 on RefCOCO+ but only 0.7 on element grounding and 0.6 on ScreenSpot. LLaVA-next-34B scores 84.8 on RefCOCO+ but only 1.7 on element grounding and 2.8 on ScreenSpot. GPT-4V scores 16.2 on ScreenSpot and 0.2 on element grounding. The authors explicitly state that 'current mllms fail to handle grounding tasks in complex gui scenarios,' identifying a reproducible condition (GUI grounding in web/app interfaces) under which these models produce severely degraded output relative to their general grounding capability.
The paper does not isolate whether the failure is due to the visual layout of GUIs, the small size of target elements, or the domain shift from natural images to screenshots; it attributes the gap to the models lacking GUI-specific training data.