IC-732Off-the-shelf PyTorch ResNet classifiers are better calibrated than fine-tuned U-Net classifiers at high noise levels in the diffusion reverse process

Jiajun Ma, Tianyang Hu, Wenjia Wang, Jiacheng Sun

SourceElucidating the design space of classifier-guided diffusion generation

The paper measures the expected calibration error (ECE) of two types of classifiers throughout the 250-step DDPM reverse process. At low noise (steps 0–50), the fine-tuned U-Net classifier shows lower ECE, reflecting its robustness to mild Gaussian noise. However, as noise magnitude increases beyond step 50, the off-the-shelf ResNet achieves consistently lower ECE than the fine-tuned classifier. This indicates that training a classifier on highly noisy, low signal-to-noise samples does not improve its calibration, and that the off-the-shelf ResNet's calibration degrades more gracefully under increasing noise.

Evidence

correlational

Caveat

The specific ECE values for the fine-tuned vs. off-the-shelf comparison are shown only in Figure 1 (curves over 250 steps); no tabulated numbers are printed in the text for this particular comparison.

Model

ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN

Datasets

ImageNet-1k / ImageNet / ImageNet-1k-val / ImageNet-Val [eval]

Methods

Expected Calibration Error / Integral Calibration Error (ECE) [primary]

Extraction

automatic-extraction