On the SpatialBench benchmark, GPT-4o's average score drops from 67.0 (RGB only) to 65.7 (RGB + depth map) when depth maps are added as extra input. The degradation is driven primarily by the 'existence' sub-task, which falls from 70.6 to 61.8, while 'counting' actually improves from 85.0 to 90.0. The authors explicitly state that 'even gpt-4o cannot fully understand depth maps,' indicating that the additional depth channel does not help and in some cases harms the model's spatial reasoning.
Evidence
correlational
Key metric
GPT-4o SpatialBench average: 67.0 (RGB) vs 65.7 (RGB+D); existence sub-task: 70.6 vs 61.8; counting sub-task: 85.0 vs 90.0
Caveat
The GPT-4o numbers are obtained from Cai et al. (2024), not directly measured by the authors of this paper. The degradation is modest in aggregate (1.3 points) and is not uniform across sub-tasks.