The paper computes the first-step DPO gradients with respect to GPT-2's MLP-value weight matrices and measures how much of the gradient is explained by the toxic subspace identified via SVD (ratio of Frobenius norms). The correlation is substantially above a random-matrix baseline, confirming that DPO moves weights in the toxic subspace direction. The correlation is stronger in later layers (justifying editing higher layers) and increases with sample size (8, 32, 128 samples), consistent with the theoretical prediction that DPO needs large samples to average out noise. Figure 6 further shows that DPO and PROFS exhibit similar incremental layer-wise contributions to token probability changes.
Evidence
correlational
Caveat
Analysis is on GPT-2 Medium only. The correlation is measured at the first DPO step only, not over the full training trajectory. The logistic model assumption in the theoretical derivation simplifies the actual transformer architecture.