SourceConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruning
Using WANDA importance scores computed on target-concept versus reference prompts, the authors identified 'skilled neurons' in the second layer of the FFN blocks of SD v1.5's UNet. An ablation across FFN-1, FFN-2, cross-attention value, and CLIP text-encoder layers shows FFN-2 is the most effective pruning target. The skilled neurons span less than 3% of the FFN weight matrix (approximately 0.12% of total model parameters). Pruning these neurons erases the target concept (e.g., Van Gogh style) while preserving the object, whereas pruning unskilled neurons removes the object but retains the style, confirming disentanglement. The authors also note that t-hat = 10 early denoising timesteps suffice, suggesting style and object features form early in the denoising trajectory.