SourceEntropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors
On the Waterbirds benchmark, where background is strongly correlated with class in training but anti-correlated in test, the pre-trained ResNet-50-BN exhibits a shortcut: its predictions are driven by background rather than the bird. The paper shows that within the worst-performing group, samples with entropy below the first quartile have lower prediction accuracy than samples in higher entropy intervals, meaning the most 'confident' predictions are the most likely to be wrong. Grad-CAM visualizations confirm that wrong low-entropy predictions focus on the background while correct ones focus on the bird. Applying entropy-based TTA (TENT) to this model degrades worst-group accuracy from 64.90% to 54.14%, below the pre-adaptation level.