IC-673Trained depthwise convolutional kernels in DS-CNN architectures converge to identifiable DoG-like patterns, with over 95% of ConvNeXtV2 and over 90% of ConvNeXt filters classifiable into a small set of clusters
Zahra Babaiee, Peyman Kiasari, Daniela Rus, Radu Grosu
The authors applied unsupervised autoencoder-based clustering to millions of depthwise convolutional kernels extracted from pre-trained DS-CNN models (ConvNeXt, ConvNeXtV2, HoNet, MoGANet, ConvMixer, EfficientNet, MobileNetV3, MixNet, MNASNet, ReplkNet-XL). The kernels converge into approximately ten identifiable clusters resembling difference-of-Gaussian (DoG) functions, their first and second-order derivatives, and cross-shaped patterns. The patterns persist across all layers, with the composition shifting: DoG patterns dominate early layers while cross-shaped patterns become predominant in the final convolutional layer. The degree of convergence varies by architecture, being highest in ConvNeXtV2 (up to 98.16%) and lowest in ConvMixer (56.64%).
The ConvMixer model shows the weakest convergence (56.64%) with noisier kernels. The 3x3 kernels (MobileNetV3) could not be well-fit by the autoencoder and required a separate k-means method. The authors note the cause of cross-shaped filters remains uncertain.