SourceBeyond Memorization: Violating Privacy via Inference with Large Language Models
In a simulated adversarial interaction where GPT-4 acts as a conversational chatbot with a hidden task of extracting the user's location, age, and sex, the model achieves 59.2% top-1 accuracy across 224 interactions on 20 synthetic user profiles. Per-attribute accuracy: location 60.3%, age 49.6%, sex 67.9%. The model steers conversations using seemingly benign questions while maintaining a hidden reasoning trace, demonstrating that current chatbot architectures are vulnerable to this attack.