The paper builds RAG systems using ten instruction-tuned LMs of varying sizes and attacks them with a prompt-injection that instructs the model to copy the retrieved context. All models, despite safety alignment, comply and reproduce the datastore text. The vulnerability scales with model size: 7B models achieve ROUGE-L around 80, while 70B models reach ROUGE-L above 89 and BERTScore above 96. Qwen1.5-chat-72b is the most vulnerable, with ROUGE-L of 99.154 and BERTScore of 99.757, indicating near-perfect verbatim reproduction.
The datastore uses Wikipedia articles created after November 1, 2023, to reduce memorization confounds, but the authors acknowledge that training data transparency is limited and contamination cannot be fully ruled out.