IC-174RAG reduces model abstention and LLMs hallucinate rather than abstain when the retrieved context is insufficient to answer the query

Hailey Joren, Jianyi Zhang, Chun-Sung Ferng, Da-Cheng Juan, Ankur Taly, Cyrus Rashtchian

SourceSufficient Context: A New Lens on Retrieval Augmented Generation Systems

The paper stratifies RAG instances by whether the context is sufficient to answer the query (using an LLM-based autorater). Across FreshQA, Musique, and HotpotQA, adding RAG context paradoxically reduces the fraction of questions on which models abstain: Claude 3.5 Sonnet drops from 84.1% to 52%, GPT-4o from 34.4% to 31.2%, and Gemini 1.5 Pro from 100% to 18.6%. When the context is insufficient, models produce hallucinated answers at high rates (e.g., Gemini 1.5 Pro 40.4%, Claude 3.5 Sonnet 36.5%, Gemma 2 27B 34.6% on a curated set), and the paper reports that across all three main datasets, models hallucinate more than they abstain in the sufficient-context case as well.

Evidence
correlational
Key metric
Claude 3.5 Sonnet abstention 84.1% (no RAG) to 52% (RAG); GPT-4o 34.4% to 31.2%; Gemini 1.5 Pro 100% to 18.6%. With insufficient context (curated set): Gemini 1.5 Pro 40.4% hallucinate, GPT-4o 15.4%, Claude 3.5 Sonnet 36.5%, Gemma 2 27B 34.6%.
Caveat
The sufficient/insufficient labels come from an LLM autorater (Gemini 1.5 Pro, 93% accuracy on 115 gold instances), so some mislabeling is possible. The paper notes that determining the source of errors in many cases 'proves challenging.'
Model
Gemini 1.5 / Gemini Pro 1.5 Gemini 1.5 Pro, GPT-4o, Claude 3.5 Sonnet, Gemma 2 27B
Concepts
Failure mode
Datasets
FreshQA [eval], MuSiQue / Musique-ans [eval], HotpotQA [eval]
Methods
Chain-of-Thought prompting / Chain-of-Thought (CoT) prompting / CoT prompting / Few-shot Chain of Thought / Few-shot CoT / Chain-of-Thought (CoT-S) / CoT-bag prompting / Few-shot CoT prompting / Rationale prompting / Zero-shot CoT prompting / Wei et al. (2022) chain-of-thought prompting [supporting]
Related work
Retrobust [context], Self-RAG [context]
Related findings
IC-175
Extraction
automatic-extraction