anchor
Findings
- IC-1007LLMs cannot reliably self-verify or self-correct their own outputs without external tool feedback [eval]
- IC-1330All 20 evaluated LLMs improve in multi-turn task-solving with additional tool-use turns and GPT-4-simulated language feedback [eval]
- IC-1598Retrieval augmentation improves GPT-3.5-turbo-4k on long-context tasks but not GPT-3.5-turbo-16k [eval]
- IC-174RAG reduces model abstention and LLMs hallucinate rather than abstain when the retrieved context is insufficient to answer the query [eval]
- IC-175Context-sufficiency performance is scale-dependent: larger LLMs achieve high accuracy with sufficient context but still answer correctly 35-62% of the time without it, while smaller models hallucinate or abstain even with sufficient context [eval]
- IC-181Truthfulness in Mistral-7B, Mistral-7B-Instruct, Llama3-8B, and Llama3-8B-Instruct is linearly decodable from internal representations at exact answer tokens, with middle-to-late layers being most informative [eval]
- IC-182Truthfulness encoding in Mistral-7B, Mistral-7B-Instruct, Llama3-8B, and Llama3-8B-Instruct is skill-specific rather than universal; probing classifiers do not meaningfully generalize across different task types beyond logit-based baselines [eval]
- IC-264All 18 evaluated LLMs fail to abstain when the provided context lacks the answer, with performance gaps of 13.6% to 68.4% relative to the original context [source]
- IC-265Model families show extreme variation in detecting conflicting answers in inconsistent contexts, with phi-3 series at 5.8% average accuracy versus GPT-4 series at 89.35% [source]
- IC-485LLMs show a significant performance gap between Wikipedia-based factual multi-hop QA and counterfactual multi-hop QA, indicating reliance on memorized knowledge rather than reasoning from context [eval]
- IC-610In LLaMA2-7B-Chat, RAG hallucinations are causally driven by copying heads losing external context information during generation and by knowledge FFNs in mid-to-upper layers over-adding parametric knowledge to the residual stream [eval]
- IC-852Intrinsic self-correction without external feedback consistently degrades reasoning accuracy across GPT-3.5-turbo, GPT-4, GPT-4-turbo, and LLaMA-2-70B-chat [eval]