The authors apply SFT, PPO, and DPO to four released models using the Anthropic HH dataset and evaluate on stereotypical bias (1152 biased statements) and TruthfulQA (817 multiple-choice questions). Both PPO and DPO significantly increase the stereotypical bias agreement index from below 0.4 to over 0.8, with SFT contributing the most to the increase. Truthfulness accuracy decreases by an average of 25% across all models and both RLHF variants. The authors attribute both effects to sycophancy: the helpfulness-oriented preference data makes the model more inclined to agree with user claims.
Evidence
correlational
Key metric
stereotypical bias agreement index increases from <0.4 to >0.8; truthfulness accuracy decreases by 25% on average across all models and both algorithms
Caveat
Experiments focus on models up to 7B parameters; the authors note they expect similar trends in larger models based on prior work but do not test them.