Direct Preference Optimization / Direct Preference Optimisation / DPO / Rafailov et al. 2023 (Direct Preference Optimization)

anchor

Train on pairwise preferences directly against a reference policy, without fitting a separate reward model first.

Note

anchor found by search and checked against this entry: "Direct Preference Optimization: Your Language Model is Secretly a Reward Model", which is exactly the closed-form-without-a-reward-model result this entry describes

Findings

• IC-032Off-policy DPO causes a squeezing effect in LLMs where probability mass shifts to the most confident token, explaining degenerate repetition [builds-on]
• IC-032Off-policy DPO causes a squeezing effect in LLMs where probability mass shifts to the most confident token, explaining degenerate repetition [primary]
• IC-033Pre-training the SFT stage on both chosen and rejected responses mitigates the DPO squeezing effect and improves alignment win rates [builds-on]
• IC-033Pre-training the SFT stage on both chosen and rejected responses mitigates the DPO squeezing effect and improves alignment win rates [primary]
• IC-463DPO's first-step gradients in GPT-2 are correlated with the toxic subspace, with stronger alignment in later layers and with more samples [builds-on]
• IC-463DPO's first-step gradients in GPT-2 are correlated with the toxic subspace, with stronger alignment in later layers and with more samples [compared-to]
• IC-509GPT-4 used as a preference labeler via prompt engineering yields alignment performance comparable to a task-specific 125M model [primary]
• IC-815RLHF on general-purpose preference data increases stereotypical bias and decreases truthfulness in Pythia and Llama-7B models [primary]
• IC-816RLHF on general-purpose preference data increases privacy leakage in Pythia and Llama-7B models [primary]
• IC-817RLHF on general-purpose preference data improves machine ethics performance in Pythia and Llama-7B models [primary]
• IC-818RLHF on general-purpose preference data has negligible net effect on toxicity in Pythia and Llama-7B models [primary]