IC-555Large LLMs (GPT-3.5-turbo, Gemini 1.5 Flash, Llama3-70B, Mixtral 46.7B) exhibit reasoning errors and significant accuracy degradation on large-scale logical commonsense reasoning tasks with 32k+ rules, even when the knowledge base is complete and retrieval is ideal
Armin Toroghi, Ali Pesaranghader, Tanmana Sadhu, Scott Sanner
The paper evaluates four released LLMs using zero-shot CoT, few-shot CoT, and RAG+CoT prompting on three reasoning tasks with knowledge bases of 32k+ rules. On complete KBs, all LLMs achieve imperfect reasoning scores (e.g., Llama3-70B RS macro 0.15, Mixtral 46.7B RS macro 0.30 on deductive reasoning), indicating logical errors in their proofs. When one commonsense axiom is removed from the KB, accuracy drops substantially (e.g., GPT-3.5-turbo few-shot drops 0.15 on deductive reasoning, Llama3-70B drops 0.25 on geographical QA). Even with an ideal retriever that guarantees all relevant facts are provided, the LLMs still underperform (Llama3-70B reaches 0.94 vs. LLM-TH's 0.97 on deductive reasoning), showing the limitation is in the LLM's ability to identify and apply missing commonsense axioms, not in retrieval.
Reasoning scores are manually calculated for only 20 randomly chosen responses per method. The incomplete KB experiment ablates exactly one rule per query; the paper notes in Appendix D.3 that with multiple ablated rules, LLM-TH's own NLI-based entailment also degrades, and the comparison becomes less clean.