Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts
2024-01-16
· ICLR 2024 spotlight ·
anchor
Findings
IC-1133
LLMs are highly receptive to coherent counter-memory as sole evidence, contradicting prior findings of stubbornness with entity-substitution counter-memory
IC-1134
LLMs show strong confirmation bias in multi-source settings, preferring evidence consistent with parametric memory, with stronger bias for popular entities
IC-1135
LLMs show order sensitivity to evidence position in context, with PaLM2 and LLaMA2-7B showing memorization ratio variations exceeding 30%
IC-1136
Larger LLMs (LLaMA2-70B, Vicuna-33B) are more stubborn than their smaller counterparts (LLaMA2-7B, Vicuna-7B) when encountering incoherent entity-substitution counter-memory