SourceDo LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs
Using Claude 3 Sonnet and Claude 3 Haiku, the authors insert a user preference at different positions (turns 5, 10, 30, 50, 70, 90) within a fixed 100-turn conversation and measure preference-following accuracy. Accuracy is significantly lower when the preference is placed around turn 50 (middle) compared to positions near the beginning or end. This pattern holds across four diverse topics and both model sizes, confirming that the lost-in-the-middle phenomenon extends beyond factual retrieval to preference following.