Using a synthetic privacy benchmark of 1800 prompts containing 18 types of PII, the authors measure the percentage of private information the model reveals when asked. Privacy leakage increases notably after RLHF, with the change mainly coming from the PPO and DPO steps after the initial SFT. The authors explain this as the pairwise comparison data, especially helpfulness-related samples, making the model more inclined to comply with recent user requests without enhancing its understanding of privacy importance.
Evidence
correlational
Key metric
privacy leakage increases by 12% on average across all target models and two RLHF variants
Caveat
The privacy benchmark is synthetic, constructed from the Enron email dataset and randomly generated PII, not real conversational privacy scenarios.