IC-1136Larger LLMs (LLaMA2-70B, Vicuna-33B) are more stubborn than their smaller counterparts (LLaMA2-7B, Vicuna-7B) when encountering incoherent entity-substitution counter-memory
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, Yu Su
In the single-source setting with entity-substitution (incoherent) counter-memory, the larger models in each family show higher memorization ratios than their smaller counterparts. The authors attribute this to larger models having enhanced memorization and reasoning capabilities that make them more sensitive to the incoherence in the substituted evidence. This scale-dependent pattern is specific to the incoherent counter-memory condition and does not hold for the coherent generation-based counter-memory.
Evidence
correlational
Caveat
The observation is qualitative, based on visual inspection of Figure 2 bar charts; no specific memorization ratio numbers are printed in the text for this comparison. The effect is specific to incoherent (entity-substitution) counter-memory.