The paper analyses the learning dynamics of LLM finetuning, decomposing the per-step change in predicted probabilities during SFT and DPO. For off-policy DPO, the negative gradient imposed on the rejected response is shown to create a 'squeezing effect': probability mass is squeezed from all but the most confident token prediction, making the model’s output distribution peakier. This is verified by tracking the log-likelihood of various response types during training on the Anthropic-HH and UltraFeedback datasets using Pythia and Qwen1.5 models. The confidence of the greedy-decoded (argmax) response increases rapidly during DPO while the confidence of all other tracked responses decreases, including the chosen response and its rephrases, offering an explanation for the 'repeater' phenomenon reported in prior work.
Evidence
correlational
Key metric
log-likelihood of greedy-decoded response increases from -113 to -63 over 8 epochs; confidence of chosen response decreases while argmax increases
Caveat
Analysis relies on the assumption that the relative influence of the empirical neural tangent kernel is stable during training, which is verified empirically but not proven theoretically.