SourceNoVo: Norm Voting off Hallucinations with Attention Heads in Large Language Models
The paper measures the L2 norm of each attention head at the final sequence position and finds that certain heads (voters) show reliable norm increases of up to 83% when the sequence contains a true factual proposition, even in the presence of misleading context. Two functional types emerge: type-1 voters attend to structural elements (periods, end tokens) and type-2 voters attend to token-level factual associations. A comparison of L2 norms from different hidden states (query, key, value, output projection, FFN1, FFN2) on Mistral-7B-Instruct shows that multi-headed attention head norms outperform all other representations, with head norms scoring 78.09 on TruthfulQA versus 64.38 for value norms and 47.25 for FFN1 norms. The paper attributes this to the output projection fusing truth-encoding and non-truth heads, and to the 32x higher dimensionality of post-projection representations.