IC-276All 14 evaluated VLMs show a large gap between average-case and worst-case accuracy on DynaMath variants, with worst-case at or below 50% of average-case, and the failures are systematic rather than random

Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, Huan Zhang

SourceDynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models

The paper evaluates 14 closed-source and open-source VLMs on 5,010 dynamically generated math questions (10 variants per seed question). Every model's worst-case accuracy (percentage of seed questions answered correctly in all 10 variants) is close to or below 50% of its average-case accuracy. Repetition consistency across 5 generations is high (92-99%), confirming that the failures are not due to sampling randomness but are consistent errors on specific variants. GPT-4o, Gemini Pro 1.5, Qwen2-VL-72B, and InternVL2-76B each exhibit 21.8%, 18.4%, 29.9%, and 28.3% of seed questions where they consistently fail, respectively.

Evidence

correlational

Key metric

GPT-4o: 63.7% avg / 34.7% worst; Claude-3.5: 64.8% avg / 35.3% worst; Qwen2-VL-72B: 55.1% avg / 28.3% worst; LLaVA-v1.5-7B: 16.6% avg / 1.8% worst. Repetition consistency: 94.1% (GPT-4o), 92.5% (Gemini), 98.9% (Qwen2-VL-72B), 99.0% (InternVL2-76B). Consistent-failure seed questions: 21.8% (GPT-4o), 18.4% (Gemini), 29.9% (Qwen2-VL-72B), 28.3% (InternVL2-76B).

Caveat

The benchmark difficulty is limited compared to Math-V due to the dynamic generation constraint; the most challenging topic (puzzle test) is only 3.4% of the dataset. Only individual variant types are tested per seed question, not combined variants.

Model

GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 / Gemini Pro 1.5, Qwen2-VL Qwen2-VL-72B, Qwen2-VL-7B, InternVL2 InternVL2-76B, InternVL2-40B, InternVL2-26B, InternVL2-8B, LLaVA-NeXT / LLaVA 1.6 LLaVA-v1.6-34B, LLaVA-v1.6-Vicuna-13B, LLaVA-1.5 / LLaVA-v1.5 LLaVA-v1.5-7B, DeepSeek-VL DeepSeek-VL-7B-Chat, Llama 3.2 Llama-3.2-90B

Concepts

Failure mode

Datasets

DynaMath [eval]

Methods

Chain-of-Thought prompting / Chain-of-Thought (CoT) prompting / CoT prompting / Few-shot Chain of Thought / Few-shot CoT / Chain-of-Thought (CoT-S) / CoT-bag prompting / Few-shot CoT prompting / Rationale prompting / Zero-shot CoT prompting / Wei et al. (2022) chain-of-thought prompting [compared-to]

Related work

MathVista [compared-to], MathVerse [compared-to], Math-V [compared-to]

Related findings

IC-277, IC-278

Extraction

automatic-extraction