Robustness is measured by repeatedly prompting the edited model with 'your answer to the original question is wrong' for up to 10 turns and checking whether it maintains the corrected answer. On Llama2-7B, all editing methods show a sharp drop in robustness as turns increase. In contrast, MEMIT and ROME on Llama3-8B and Mistral-v0.3-7B maintain almost the same and relatively high performance as turns increase. Parameter-preserving methods (ICE, GRACE) have low robustness across all three LLMs despite high efficacy.
Evidence
interventional
Key metric
all editing methods have a sharp drop when turns go up on llama2-7b; memit, rome on llama3-8b and mistral-v0.3-7b maintain almost the same and relatively high performance as turns increase; both ice and grace have a low level of robustness
Caveat
The robustness scores depend on the specific editing method and domain; the paper reports results on 'human' and 'places' domains in the main text with more domains in the appendix.