When the full list of books by the author is provided as in-context information (simulating a perfect retrieval system), the irrelevance rate drops to near zero (GPT-4: 0.08, GPT-3.5: 0.07). However, the satisfaction rate barely changes: GPT-4 stays at 0.24 and GPT-3.5 at 0.16. All-correctness remains very low (GPT-4: 0.08, GPT-3.5: 0.07). This shows the bottleneck is the model's ability to verify and filter constraints, not its knowledge of the book list.
Evidence
correlational
Key metric
GPT-4: p_irr 0.26 (no-context) to 0.08 (with-context), p_sat 0.24 (no-context) to 0.24 (with-context); GPT-3.5: p_irr 0.20 to 0.07, p_sat 0.16 to 0.16; all-correctness GPT-4 0.08, GPT-3.5 0.07
Caveat
The context provided is the complete and correct list from KITAB, which is an idealized retrieval setting; real retrieval systems may provide incomplete or noisy context.