IC-500GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and Llama 3.1 405B solve multi-step retrieval problems without fine-tuning, achieving near-perfect accuracy for chains of up to 5 steps
The paper introduces the retrieval problem, a task requiring chained multi-step retrieval (e.g., a=7, b=a, c=b, d=c, what is d?). Four state-of-the-art LLMs are evaluated zero-shot on 500 randomly generated questions across five formulations (equations, lives-with, kingdoms, functions, relatives) with no fine-tuning. All four models achieve near-perfect accuracy on the equations formulation for retrieval chains of up to 5 steps (d ≤ 5), and strong performance on the lives-with, kingdoms, functions, and relatives formulations at d = 5. Accuracy degrades as the chain length increases beyond 5 steps, with all models falling toward random-guessing levels at d = 13.
Evidence
correlational
Key metric
500 questions per formulation; near-perfect accuracy for d ≤ 5 (equations formulation); strong performance at d = 5 (lives-with, kingdoms, functions, relatives formulations); accuracy degrades toward random guessing at d = 13
Caveat
The paper does not report exact per-model accuracy numbers in the text; specific values are only shown in Figure 2. The evaluation uses a single sampling strategy (repeat until acceptable answer) which may inflate accuracy.