By computing the SVD of the difference between toxic and non-toxic sentence embeddings in GPT-2's MLP layers, the paper identifies a low-dimensional toxic subspace. The top singular vectors map to toxic vocabulary (e.g., profanity, slurs) when projected onto the output embedding matrix, while lower-ranked vectors correspond to context-dependent topics. This toxic subspace is not uniformly distributed across layers: editing only the higher layers (15-24) reduces toxicity to 26.83% while preserving perplexity, whereas editing the lower layers (1-10) actually increases toxicity to 74.63% and degrades perplexity. The interpretation is consistent across different data subsets and layers.
Analysis is limited to GPT-2 Medium (355M) and MLP layers only; the paper notes that self-attention layers are not explored. The toxic subspace rank is small (k=2 for GPT-2), and the interpretation relies on mapping singular vectors to top-scoring vocabulary tokens.