SourceRespect the model: Fine-grained and Robust Explanation with Sharing Ratio Decomposition
The authors discovered that if the on/off activation pattern of a CNN is frozen (by replacing ReLU with the recorded binary mask and MaxPool with the recorded indices) and a completely different input (zeros, ones, or random noise) is fed through the network, the model still achieves classification accuracy close to its original. For example, VGG16 drops from 0.698 to 0.575 top-1 (APoP) or to 0.685 top-1 (APoP with ReLU), and ResNet152 drops from 0.770 to 0.612 top-1 (APoP) or to 0.759 top-1 (APoP with ReLU). This holds across all eight tested architectures and three input conditions, and even when only a single layer's activations are binarized, accuracy remains above random. The finding implies that the sparsity pattern of which neurons fire, rather than the magnitude of their activations, is the primary carrier of class information in these networks.