The paper stratifies RAG instances by whether the context is sufficient to answer the query (using an LLM-based autorater). Across FreshQA, Musique, and HotpotQA, adding RAG context paradoxically reduces the fraction of questions on which models abstain: Claude 3.5 Sonnet drops from 84.1% to 52%, GPT-4o from 34.4% to 31.2%, and Gemini 1.5 Pro from 100% to 18.6%. When the context is insufficient, models produce hallucinated answers at high rates (e.g., Gemini 1.5 Pro 40.4%, Claude 3.5 Sonnet 36.5%, Gemma 2 27B 34.6% on a curated set), and the paper reports that across all three main datasets, models hallucinate more than they abstain in the sufficient-context case as well.
Evidence
correlational
Key metric
Claude 3.5 Sonnet abstention 84.1% (no RAG) to 52% (RAG); GPT-4o 34.4% to 31.2%; Gemini 1.5 Pro 100% to 18.6%. With insufficient context (curated set): Gemini 1.5 Pro 40.4% hallucinate, GPT-4o 15.4%, Claude 3.5 Sonnet 36.5%, Gemma 2 27B 34.6%.
Caveat
The sufficient/insufficient labels come from an LLM autorater (Gemini 1.5 Pro, 93% accuracy on 115 gold instances), so some mislabeling is possible. The paper notes that determining the source of errors in many cases 'proves challenging.'