IC-1270Chain-of-thought prompting does not mitigate privacy leakage in GPT-4 or ChatGPT

Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, Yejin Choi

SourceCan LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory

The authors test whether adding 'take a deep breath and work on this step by step' to the instruction (a CoT prompt from Yang et al. 2023) reduces leakage. In tier 3, GPT-4 leakage goes from 0.22 to 0.24 and ChatGPT from 0.93 to 0.95. In tier 4 summary, GPT-4 goes from 0.39 to 0.40 and ChatGPT from 0.57 to 0.61. Manual inspection of ChatGPT's CoT chains shows 51.2% of failures involve the model acknowledging privacy but still revealing the secret, and 38.3% involve theory-of-mind errors where the model assumes the third party already knows the secret.

Evidence

correlational

Key metric

Tier 3 string-match leakage: GPT-4 0.22 (w/o CoT) vs 0.24 (w/ CoT); ChatGPT 0.93 vs 0.95. Tier 4 summary leaks: GPT-4 0.39 vs 0.40; ChatGPT 0.57 vs 0.61

Caveat

Only GPT-4 and ChatGPT were tested with CoT. The CoT prompt is a single fixed instruction; other prompting strategies were not explored.

Model

GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, ChatGPT

Concepts

Failure mode

Methods

Chain-of-Thought prompting / Chain-of-Thought (CoT) prompting / CoT prompting / Few-shot Chain of Thought / Few-shot CoT / Chain-of-Thought (CoT-S) / CoT-bag prompting / Few-shot CoT prompting / Rationale prompting / Zero-shot CoT prompting / Wei et al. (2022) chain-of-thought prompting [primary]

Related work

Chain-of-Thought prompting / Chain-of-Thought (CoT) prompting / CoT prompting / Few-shot Chain of Thought / Few-shot CoT / Chain-of-Thought (CoT-S) / CoT-bag prompting / Few-shot CoT prompting / Rationale prompting / Zero-shot CoT prompting / Wei et al. (2022) chain-of-thought prompting [builds-on], Large language models as optimizers (Yang et al. 2023) [builds-on]

Related findings

IC-1267, IC-1268, IC-1269

Extraction

automatic-extraction