MAE-B/16

image · generative · anchor

Findings

• IC-290Zeroing out or doubling specific FFN neurons identified by the neuron path method causes significant accuracy changes in ViT and MAE models
• IC-291ViT-B/16 and MAE-B/16 exhibit nearly inverted distributions of knowledge neurons across layers despite identical architecture and training data
• IC-292Neuron paths in ViT-B/16 show class-specific neuron clustering and semantic similarity between image categories

Shared mechanisms

• Depth-dependent structurealso in Baichuan 2, BERT, BLIP-2, BLOOM, Chameleon, CLIP / CLIP-ViT (LC), DeepFloyd IF, DeiT-III, DINO, DINOv2, Falcon, Gemma, Gemma 2, GPT-2, GPT-J, GPT-NeoX-20B, Griffin, I3D, Idefics, InstructBLIP, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama 3.2, Llama-3.2-3B, LLaVA, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Phi, MAE, Mamba, MiniGPT-4, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, mPLUG-Owl, MPT, MultiBERTs, MViT V2, OLMo / OLMo base, OpenCLIP, OPT, Phi-2, Pythia, Qwen 2, Qwen2-VL, Qwen2.5, RoBERTa / RoBERTa-L, RWKV, SALMONN, SAM, SlowFast, Stable Diffusion, Swin Transformer, TerraMind, TimesFormer, TSM, Uniformer, Vicuna, VideoMAE, ViT, X3D, Yi
• Distance preservationalso in CLIP / CLIP-ViT (LC), CoPlace, DINO, DINOv2, Gemma, Gemma 2, ImageBind, LanguageBind, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama-3.2-3B, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Med, MAE, OpenCLIP, Pythia, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, SigLIP, SLIP, TerraMind, ViT
• Explanation faithfulnessalso in BakLLaVA, CF2, Claude 3, Claude 3.5, CLIP / CLIP-ViT (LC), DRUM, Fuyu, GEM, Gemini 1.5 / Gemini Pro 1.5, Gemma 2, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4o, GPT-J, Idefics, Llama 3.1, LLaVA-NeXT / LLaVA 1.6, MobileNetV2, mPLUG-Owl3, OpenFlamingo, PGExplainer, Pythia, RCExplainer, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, SigLIP, SigLIP-2, Stable Diffusion, TAGExplainer, ViT