IC-1016ImageNet-pretrained ResNet-50 backbone exhibits shortcut bias toward background features when adapted to bird classification via a new readout layer

Katherine Hermann, Hossein Mobahi, Thomas FEL, Michael Curtis Mozer

SourceOn the Foundations of Shortcut Learning

The authors freeze the ImageNet-pretrained ResNet-50 backbone and train a randomly initialized readout layer for 15 epochs on the Waterbirds dataset. They then measure bird classification accuracy on incongruent probes (where bird and background labels disagree) under six background availability manipulations: bird size scaling, background patch removal, color-to-grayscale conversion, low-pass filtering, uniform noise, and Perlin noise. The pretrained backbone shows the same pattern as from-scratch models: reducing background availability (larger bird, noisier or filtered background) shifts the model toward using the bird, while a clear informative background pulls the model toward the background. The authors note that pretraining modulates feature availability, attributing this to learned invariances.

Evidence

correlational

Caveat

The readout layer is randomly initialized and trained for only 15 epochs; the authors note the pretraining effect 'may be attributed to the learned invariances during the pretraining process' without isolating the mechanism. Results are shown in a supplementary figure (B.9) without specific accuracy values printed in the text.

Model

ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN ResNet50

Concepts

Shortcut

Datasets

Waterbirds [eval]

Related work

Wolff & Wolff (2022) [context]

Extraction

automatic-extraction