IC-259Fine-tuned DNNs approach human accuracy on VPT-basic but fail on VPT-strategy, revealing reliance on a brittle feature-based shortcut (object size and location) rather than line-of-sight estimation

Drew Linsley, Peisen Zhou, Alekh Karkada Ashok, Akash Nagaraj, Gaurav Gaonkar, Francis E Lewis, Zygmunt Pizlo, Thomas Serre

SourceThe 3D-PC: a benchmark for visual perspective taking in humans and machines

After fine-tuning on VPT-basic, 97% of TIMM DNNs exceeded human accuracy on depth order and 4 reached human accuracy on VPT-basic. However, on the VPT-strategy task, which holds the scene camera fixed and moves the target objects to break feature-based correlations, humans scored 87% while the best DNN (Swin Transformer ImageNet-21k) scored only 66%. Smooth Gradients attribution maps confirmed that DNNs attend to the positions of the green camera and red ball, and the precision of this localization increased with VPT-basic accuracy. The authors conclude DNNs rely on object size and location as a shortcut rather than estimating line-of-sight as humans do.

Evidence

correlational

Key metric

97% of DNNs exceeded human accuracy on depth order after fine-tuning; 4 DNNs reached human accuracy on VPT-basic; humans 87% on VPT-strategy; best DNN (Swin Transformer ImageNet-21k) 66% on VPT-strategy

Caveat

The authors note they were not able to characterize exactly how DNNs use object features for VPT, which may be critical for developing next-generation models.

Model

Swin Transformer

Concepts

Shortcut

Related findings

IC-257, IC-258

Extraction

automatic-extraction