Across 10 open-source and proprietary LLMs, preference-following accuracy in zero-shot settings drops steeply as the number of conversation turns between the stated preference and the query increases. At 5 turns accuracy falls from roughly 80% to below 30%, and at 300 turns it approaches zero for all models. Even at just 10 turns (~3k tokens), GPT-4o, Claude 3.5 Sonnet, and Gemini-1.5-Pro all score only 0.07. The dominant error type in zero-shot is preference-unaware violation, meaning the model simply does not track the earlier preference.
Evidence
correlational
Key metric
accuracy drops from approximately 80% to below 30% at 5 turns; at 300 turns accuracy falls close to zero; at 10 turns zero-shot: GPT-4o 0.07, Claude 3.5 Sonnet 0.07, Gemini-1.5-Pro 0.07, O1-preview 0.50, O3 0.71
Caveat
The o-series models may involve additional test-time compute due to their thinking phase, making direct comparison with other models potentially unfair.