IC-510Qwen2-7B and Llama3-8B score near-random on textual temporal reasoning tasks while Qwen2-72B, Llama3-70B, and GPT-4o achieve near-perfect accuracy, showing temporal reasoning in LLMs is scale-dependent and emerges only above ~70B parameters
Lei Li, Yuanxin Liu, Linli Yao, Peiyuan Zhang, Chenxin An, Lean Wang, Xu Sun, Lingpeng Kong, Qi Liu
The paper probes the temporal reasoning ability of LLMs by presenting textual frame captions (generated by GPT-4o) alongside multiple-choice temporal questions covering order, attribute change, temporal referring, and temporal grounding. Small LLMs (Qwen2-7B, Llama3-8B) score near random on the attribute task (67.4% and 52.8%) and well below ceiling on referring and grounding (68.0-83.1%). In contrast, large LLMs (Qwen2-72B, Llama3-70B, GPT-4o) score above 97% on referring and grounding. Meanwhile, simple LSTM probes trained on the visual features of released video LLMs (LongVA-7B, ViLA-8B) achieve 88.9-96.6% accuracy, demonstrating the visual encoder captures temporal information adequately. The authors conclude the LLM backbone, not the visual encoder, is the primary bottleneck for temporal reasoning in video LLMs.
Evidence
correlational
Key metric
Qwen2-7B: attribute 67.4, referring 70.7, grounding 68.0, order 70.7; Llama3-8B: attribute 52.8, referring 83.1, grounding 72.5, order 80.8; Qwen2-72B: attribute 69.2, referring 99.7, grounding 99.1, order 99.8; Llama3-70B: attribute 65.3, referring 97.6, grounding 98.2, order 92.2; GPT-4o: attribute 69.6, referring 99.8, grounding 99.4, order 99.8; LSTM probe on LongVA-7B visual features: 88.9, 92.7, 98.2, 96.4; LSTM probe on ViLA-8B visual features: 92.8, 96.1, 94.9, 96.6
Caveat
The probing uses synthesized videos with controlled, simple temporal aspects (e.g., two or three concatenated clips, brightness changes). The attribute task has only 2 classes, so random baseline is 50%. The authors note the temporal concept scope is limited to four dimensions and that motion-related changes still need further investigation.