RedditQA

Findings

• IC-177GPT-4o mini, GPT-4o, and Llama-3-8B all over-rely on incorrect external context, producing wrong answers at high rates when the context conflicts with their internal knowledge [eval]
• IC-178Self-guided confidence reasoning (SCR) outperforms rule-based confidence reasoning (RCR) for GPT-4o and GPT-4o mini, but RCR outperforms SCR for Llama-3-8B [eval]
• IC-179GPT-4o mini, GPT-4o, and Llama-3-8B all calibrate confidence in their internal answers significantly better than confidence in external contexts [eval]
• IC-180GPT-4o-mini's resistance to incorrect context depends on the position of the context relative to the question in the prompt [eval]