SourceKnowledge Localization: Mission Not Accomplished? Enter Query Localization!
The authors identify high-attention-score positions (knowledge synapses) and intervene by suppressing or enhancing the attention scores at those positions. Suppressing knowledge synapses significantly decreases the activation values of the target knowledge neurons and reduces the model's answer probability, while neighbor KNs and randomly selected non-KNs show minimal change. Enhancing knowledge synapses has a smaller effect than suppressing, suggesting the attention module is already performing selection and further enhancement leads to saturation. This shows the attention module acts as a selector of which KNs to activate for a given query, rather than a storage site.