Zero-shot Chain-of-Thought / Wei et al. 2022 (Chain of Thought) / Zero-shot chain-of-thought prompting

Findings

• IC-1157GPT-4 and other LMs show a large gap between rule induction and rule application, with task accuracy dropping to near zero on MiniScan when the LM itself applies its own proposed rules [compared-to]
• IC-1177LLM-generated misinformation is harder for LLM detectors to detect than human-written misinformation with the same semantics [primary]
• IC-384Llama-3-8B-Instruct shows larger absolute gains from prompt optimization than the stronger Gemma-2-9B-IT, indicating that prompt-optimization benefit is inversely related to base model capability [compared-to]
• IC-433LLMs exhibit a non-monotonic ID-OOD performance gap (generalization valley) that peaks at intermediate task complexity [primary]
• IC-434The critical complexity at which LLMs over-rely on memorization shifts to higher task difficulty as model size increases [primary]
• IC-435Mistral-7B employs a less efficient algorithmic strategy (O(n²)) than Llama-3-8B (O([n², n³])) on probe tasks with multiple solution complexities [primary]
• IC-487Including sub-questions in the prompt improves LLM performance on multi-hop QA tasks [builds-on]
• IC-500GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and Llama 3.1 405B solve multi-step retrieval problems without fine-tuning, achieving near-perfect accuracy for chains of up to 5 steps [context]
• IC-555Large LLMs (GPT-3.5-turbo, Gemini 1.5 Flash, Llama3-70B, Mixtral 46.7B) exhibit reasoning errors and significant accuracy degradation on large-scale logical commonsense reasoning tasks with 32k+ rules, even when the knowledge base is complete and retrieval is ideal [primary]