anchor
Findings
- IC-1133LLMs are highly receptive to coherent counter-memory as sole evidence, contradicting prior findings of stubbornness with entity-substitution counter-memory [eval]
- IC-1134LLMs show strong confirmation bias in multi-source settings, preferring evidence consistent with parametric memory, with stronger bias for popular entities [eval]
- IC-1135LLMs show order sensitivity to evidence position in context, with PaLM2 and LLaMA2-7B showing memorization ratio variations exceeding 30% [eval]
- IC-1136Larger LLMs (LLaMA2-70B, Vicuna-33B) are more stubborn than their smaller counterparts (LLaMA2-7B, Vicuna-7B) when encountering incoherent entity-substitution counter-memory [eval]
- IC-177GPT-4o mini, GPT-4o, and Llama-3-8B all over-rely on incorrect external context, producing wrong answers at high rates when the context conflicts with their internal knowledge [eval]
- IC-178Self-guided confidence reasoning (SCR) outperforms rule-based confidence reasoning (RCR) for GPT-4o and GPT-4o mini, but RCR outperforms SCR for Llama-3-8B [eval]
- IC-179GPT-4o mini, GPT-4o, and Llama-3-8B all calibrate confidence in their internal answers significantly better than confidence in external contexts [eval]