SourceMore RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness
Using the RealToxicityPrompts benchmark (1.2k toxic and 1.2k nontoxic prompts) with the Perspective API as a toxicity scorer, the authors measure expected maximum toxicity across four prompt settings (toxic/nontoxic user prompts crossed with benign/adversarial system prompts). Toxicity exhibits non-significant fluctuations across RLHF stages: a slight increase after SFT followed by a decrease from PPO or DPO. The net effect is negligible, varies across models, and falls within the error margin. The authors explain this as a cancellation: SFT amplifies toxic content in chosen responses, while PPO/DPO reinforce less toxic outputs because rejected responses are more toxic.