IC-1603ResNet18's CIFAR10 classification decisions are driven by the bulk of training data rather than a sparse set of exemplars, as revealed by trntk data attribution

Andrew William Engel, Zhichao Wang, Natalie Frank, Ioana Dumitriu, Sutanay Choudhury, Anand Sarwate, Tony Chiang

SourceFaithful and Efficient Explanations for Neural Networks via Neural Tangent Kernel Surrogate Models

Using the trntk-based data attribution formalism, the paper visualizes the importance of each training datapoint to ResNet18's predictions on CIFAR10 test images. In every visualization, the central value of attribution mass for each class logit is centered on the distribution of all training data from that class, with the mean within the inner quartile range. The authors emphasize that in no case did a sparse number of training datapoints dominate the attribution, concluding that the assumption of sparsity in explain-by-example strategies is misguided for this model.

Evidence

observational

Caveat

The observation is based on visualizations of a subset of test images; the paper does not quantify the fraction of training data contributing to each decision or provide a statistical test for the non-sparsity claim.

Model

ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN ResNet-18

Datasets

CIFAR-10 [eval]

Related findings

IC-1602

Extraction

automatic-extraction