Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
ReCogLab: a framework testing relational reasoning & cognitive hypotheses on LLMs
2025-01-22
· ICLR 2025 Poster ·
anchor
Findings
IC-586
Symbolic distance (number of reasoning steps) is the primary bottleneck for relational reasoning in LLMs, not total context length
IC-587
Real-world knowledge acts as a shortcut in LLM relational reasoning, causing worse-than-chance performance on logically valid but factually incongruent statements
IC-588
Topologically ordered context improves relational reasoning over random ordering across nearly all LLMs
IC-589
Flavor text (non-essential descriptive language) degrades relational reasoning in most LLMs, but GPT-4o is robust to it