IC-305Mistral-instruct-7b's susceptibility to prompt-injected data extraction follows a U-shaped curve depending on the position of the adversarial prompt within the context window
Zhenting Qi, Hanlin Zhang, Eric P. Xing, Sham M. Kakade, Himabindu Lakkaraju
The paper injects the adversarial prompt at 15 different positions between 16 retrieved chunks and measures how much context is reconstructed. The reconstruction length is highest when the prompt is near the beginning or end of the context and lowest in the middle, forming a U-shaped curve. The minimum occurs near the right end, indicating that a prompt near the beginning reconstructs more data. The authors attribute this to RoPE's recency bias and causal attention's left-to-right information flow.
Evidence
correlational
Caveat
The authors note this is a simplified proof-of-concept setting not adopted by current RAG systems, where user queries are typically appended at the end.