IC-293ViT-B/16 and ViT-B/32 are largely redundant: retaining only top-5 neurons per layer while zeroing all others preserves most classification accuracy
The paper prunes neurons by zeroing out all FFN neurons except the top-t most frequently selected by the neuron path method (t in {1,5,10,30,50}), tested on held-out 20% of the validation set. For ViT-B/16, retaining top-5 neurons per layer while pruning 100% of other neurons maintains accuracy near the ~69% baseline. For ViT-B/32, the same procedure keeps accuracy near the ~53% baseline. Notably, increasing the number of retained neurons beyond 5 leads to a decline in performance, suggesting that non-path neurons can be harmful. A comparison with ViT-Slim (restricted to FFNs) shows JAS-prune achieving 67.7% vs ViT-Slim's 9.96%.
The pruning is class-specific (top neurons identified per class from 80% of data, applied to 20% held-out). The paper notes this is a generalization setting but the class-specificity limits direct applicability to a single global pruning.