Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Reasoning Score
anchor
Findings
IC-555
Large LLMs (GPT-3.5-turbo, Gemini 1.5 Flash, Llama3-70B, Mixtral 46.7B) exhibit reasoning errors and significant accuracy degradation on large-scale logical commonsense reasoning tasks with 32k+ rules, even when the knowledge base is complete and retrieval is ideal
[eval]