The paper evaluates 16 VidLMs, 2 ILMs (CLIP, BLIP-2), and 2 unimodal baselines (GPT-2, OPT) on five temporal reasoning tests (action counting, situation awareness, change of state, rare actions, spatial relations) using pairwise ranking accuracy. In the majority of tasks, VidLMs deliver performance levels closely resembling those of ILMs. In counting, situation awareness, and change of state, many VidLMs do not show notably higher performance than the random baseline (23.8 avg). The best ILM (BLIP-2, 57.5 avg P+T) matches or exceeds most VidLMs, with only Video-LLaMA (57.3) and InternVideo (54.2) approaching it.
The paper notes that foil generation introduces plausibility biases that may inflate unimodal and ILM scores in situation awareness and spatial relations. The evaluation uses a limited number of frames (k=4 or 8) for most models, which may further limit temporal reasoning.