When the model is asked to first retrieve all books by the author and then filter by constraint (self-context template), the irrelevance rate increases rather than decreases. GPT-4 goes from 0.26 (no-context) to 0.33 (self-context), and GPT-3.5 from 0.20 to 0.44. The first step of the chain (listing all books) already has high irrelevance (0.42 for GPT-4, 0.47 for GPT-3.5), and the model cannot recover from these fabricated titles in the filtering step. Qualitative examples show the model generating irrelevant titles specifically to satisfy the constraint.
Evidence
correlational
Key metric
GPT-4: p_irr 0.26 (no-context) to 0.33 (self-context); GPT-3.5: p_irr 0.20 to 0.44; first-step irrelevance 0.42 (GPT-4) and 0.47 (GPT-3.5)
Caveat
The self-context condition uses a maximum token length of 3000, which may affect generation quality differently from the 400-token no-context condition.