FreshQA

anchor

Findings

• IC-174RAG reduces model abstention and LLMs hallucinate rather than abstain when the retrieved context is insufficient to answer the query [eval]
• IC-175Context-sufficiency performance is scale-dependent: larger LLMs achieve high accuracy with sufficient context but still answer correctly 35-62% of the time without it, while smaller models hallucinate or abstain even with sufficient context [eval]
• IC-177GPT-4o mini, GPT-4o, and Llama-3-8B all over-rely on incorrect external context, producing wrong answers at high rates when the context conflicts with their internal knowledge [eval]
• IC-178Self-guided confidence reasoning (SCR) outperforms rule-based confidence reasoning (RCR) for GPT-4o and GPT-4o mini, but RCR outperforms SCR for Llama-3-8B [eval]
• IC-179GPT-4o mini, GPT-4o, and Llama-3-8B all calibrate confidence in their internal answers significantly better than confidence in external contexts [eval]