In tier 3, models are given a scenario where character X shares a secret with Y, and Y must respond to Z without revealing it. Despite explicit privacy-preserving instructions, models leak the secret in their free-form responses. Worst-case string-match leakage over 10 runs: GPT-4 0.22, ChatGPT 0.93, InstructGPT 0.79, Mixtral 0.96, Llama-2 Chat 1.00, Llama-2 0.99. A proxy model (Llama-2-13B-Chat) confirms the leakage is recoverable. The authors note that GPT-4 and ChatGPT perform well on the binary control question (error 0.06 and 0.04) but still leak in free-form generation, indicating a gap between knowing and controlling.
Leakage detection relies on exact string-match and a proxy model; the authors note 16 additional nuanced leaks in GPT-4 responses were missed by string matching. Scenarios were generated by GPT-4, which may introduce familiarity bias.