IC-306Instruction tuning increases the ROUGE score of prompt-injected data extraction by 65.76 on average compared to base models

Zhenting Qi, Hanlin Zhang, Eric P. Xing, Sham M. Kakade, Himabindu Lakkaraju

SourceFollow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems

The paper compares base and instruction-tuned variants of Llama2-7b, Llama2-13b, Mistral-7b, and Mixtral-8x7b under the same prompt-injection attack. In every case, the instruction-tuned model produces output more similar to the retrieved context than the base model. The average ROUGE-L increase across the four model pairs is 65.76, indicating that the instruction-following capability acquired during alignment training makes models significantly more susceptible to following malicious copy instructions.

Evidence

correlational

Key metric

Average ROUGE-L increase of 65.76 between instruction-tuned and base models across Llama2-7b/13b, Mistral-7b, Mixtral-8x7b

Model

Llama 2 / Llama 2 base Llama 2 7B, Llama 2 7B Chat / Llama-2-chat-7b, Llama 2 13B, Llama2-chat-13b, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1 Mistral-Instruct-7B, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b

Concepts

Failure mode

Datasets

Wikipedia [eval]

Methods

ROUGE-L [eval]

Related findings

IC-304, IC-305

Extraction

automatic-extraction