SourceLearning Dynamics of LLM Finetuning
Inspired by the analysis of the squeezing effect, the paper proposes a method to mitigate it by augmenting the SFT dataset with (prompt, rejected response) pairs before running DPO. This pulls up the probability of the rejected response region, so the subsequent DPO negative gradient is imposed on a less unlikely region, weakening the squeezing effect. The method is evaluated on the Anthropic-HH dataset using Qwen1.5-1.8B. The proposed 'extend' method shows a slower decay of other responses during DPO and a slower increase in the argmax confidence. In pairwise comparisons against a baseline SFT-then-DPO pipeline using GPT-3.5-Turbo and Claude3-Haiku as judges, the extended method achieves a win rate of 0.6928 and 0.6045 respectively after 4 epochs of DPO, indicating improved alignment performance.