IC-411Llama2-7B, Llama3-8B, and Mistral-v0.3-7B do not reason with edited knowledge in multi-hop questions, as editing methods mostly underperform pre-edit portability scores
Baixiang Huang, Canyu Chen, Xiongxiao Xu, Ali Payani, Kai Shu
Portability is measured by whether the model can answer multi-hop questions (2-6 hops) that depend on the edited single-hop fact. The paper finds that except for ICE which may bring marginal improvement, all other editing techniques underperform the pre-edit portability scores. There is a sharp decrease from single-hop to multi-hop for all methods. The authors hypothesize that the LLMs directly memorize answers to multi-hop questions rather than reasoning from the single-hop knowledge, which is why editing the single-hop fact does not propagate.
Evidence
interventional
Key metric
except that ice may bring marginal improvement to the pre-edit performance, the other knowledge editing techniques even mostly underperform pre-edit portability scores; a sharp decrease for all the editing methods from single-hop to multi-hop
Caveat
The memorization hypothesis is stated as a hypothesis, not a confirmed mechanism; pre-edit portability scores are not zero for 2-6 hops.