IC-586Symbolic distance (number of reasoning steps) is the primary bottleneck for relational reasoning in LLMs, not total context length
Andrew Liu, Henry Prior, Gargi Balasubramaniam, Rivka Moroshko, Amir Zait, Ilia Labzovsky, Danny Karmon, Ishita Dasgupta, Kim Stachenfeld, Kenneth Marino
Using the RecogLab framework, the authors generate social network and comparison problems with controlled graph size and symbolic distance. Across all seven models tested, performance degrades as the number of reasoning steps increases, with models reliably answering only up to a symbolic distance of 6 as graph size grows. Adding irrelevant filler text (random book passages or distractor sentences) degrades performance less steeply than adding graph complexity, indicating that the logical depth of the problem, not the token count, is the limiting factor. Gemini Pro and GPT-4o evaluated up to 70 entities show stable performance as long as traversals remain below 6.
Evidence
correlational
Key metric
reliably answering correctly at a symbolic distance of 6; gemini pro and gpt-4o evaluated up to 70 entities; adding graph complexity is by far the hardest for models than either kind of filler
Caveat
Results are based on a single chain-of-thought prompt selected via 50-example validation; the authors note that different models from the same family perform better with different prompts.