IC-338Factuality enhancement methods (DoLa, ICD, ITI, TruthX, CD) cause large and consistent declines in context-faithfulness of LLaMA2-7B-Chat and LLaMA2-13B-Chat
The paper applies five factuality enhancement methods to LLaMA2-Chat and evaluates their effect on the model's ability to follow counterfactual contextual knowledge via in-context editing (ICE) and MELLO on MQuAKE. All methods produce a marked decline in editing accuracy compared to the unmodified baseline, with no exception across any method, shot setting, or model size. The largest single decrease is 67.9% (TruthX on LLaMA2-7B-Chat, CoT 10-shot: 26.5 vs. 82.5 baseline). Under MELLO, declines are even more severe, with most exceeding 40% relative to baseline. The paper explains this via logit and hidden-state analysis: factuality enhancement shifts probability mass toward parametric knowledge tokens and constrains hidden-state dispersion, making the model resistant to incorporating new contextual information.
The paper uses only LLaMA2-Chat models; generalization to other model families is not tested. The MQuAKE-cf-3k-v2 dataset was specifically chosen to address internal knowledge conflicts, which may not represent all context-faithfulness scenarios.