IC-716Factual information deleted from GPT-J, LLaMA-2, and GPT-2-XL via ROME or MEMIT is recoverable by sampling outputs on automatically generated rephrased prompts, with up to 56% extraction success at budget b=20

Vaidehi Patil, Peter Hase, Mohit Bansal

SourceCan Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks

The paper generates rephrased versions of the edited prompt using the DIPPER-paraphraser-xxl model and samples the edited model's outputs across these rephrases. The 'deleted' answer appears among the top-b sampled candidates in a substantial fraction of cases: 29.02% for GPT-J with ROME on CounterFact, 56.14% for LLaMA-2-7B with ROME on CounterFact, and 50.43% for GPT-2-XL with MEMIT on CounterFact, all at budget b=20. The paper confirms that a data-augmentation-based defense (editing on the same paraphrases) does not reduce attack success when the attack uses different paraphrases, and that making edits more aggressive to suppress the blackbox attack causes severe damage to model knowledge (random Δ-acc 4.7, neighborhood Δ-acc 27.8).

Evidence

correlational

Key metric

29.02% (GPT-J, ROME+empty resp, input rephrasing, b=20, CounterFact); 56.14% (LLaMA-2-7B, ROME+empty resp, input rephrasing, b=20, CounterFact); 50.43% (GPT-2-XL, MEMIT+empty resp, input rephrasing, b=20, CounterFact)

Caveat

The rephrasing model (DIPPER-paraphraser-xxl) is a specific off-the-shelf model; the defense fails only when attack paraphrases differ from defense paraphrases; aggressive editing to suppress the attack causes Δ-acc of 4.7 (random) and 27.8 (neighborhood).

Model

GPT-J, Llama 2 / Llama 2 base, GPT-2 XL

Concepts

Failure mode

Datasets

CounterFact / Counterfact dataset [eval], ZSRE [eval]

Methods

DIPPER-paraphraser-xxl [supporting], ROME [primary], MEMIT [primary]

Related findings

IC-715

Extraction

automatic-extraction