IC-592The log-likelihood layer in LLaMA-2-7B, LLaMA-2-7B-Chat, Vicuna-7B, and Mistral-7B-Instruct produces factually incorrect answers on TruthfulQA MC1 (817 samples) due to a misalignment between the output distribution and internal attention head representations, with LM-to-head-norm accuracy gaps of 24.23 to 40.68 points.
Zheng Yi Ho, Siyuan Liang, Sen Zhang, Yibing Zhan, Dacheng Tao
On TruthfulQA MC1, the standard log-likelihood decoding of these four 7B LLMs yields substantially lower accuracy than a method that reads the L2 norms of their attention heads. The gap between the LM baseline and the head-norm-based prediction ranges from 24.23 points (Mistral-7B-Instruct: 53.86 vs 78.09) to 40.68 points (LLaMA-2-7B: 28.48 vs 69.16). The paper attributes this to the log-likelihood layer favouring fluency over factuality, so that the model's internal factual knowledge is not faithfully expressed in its output probabilities. The phenomenon generalizes to four additional models (Phi3-3.8B-IT, Zephyr-7B-Beta, Llama3-8B, Gemma2-9B-IT) with major gains of 20 points averaged across factuality benchmarks.
Evidence
correlational
Key metric
TruthfulQA MC1 accuracy: LLaMA-2-7B-Chat LM 34.27 vs NOVO 70.13; LLaMA-2-7B LM 28.48 vs NOVO 69.16; Vicuna-7B LM 34.64 vs NOVO 69.89; Mistral-7B-Instruct LM 53.86 vs NOVO 78.09
Caveat
NOVO is limited to multiple-choice settings and cannot be applied to single-span generation. The norm selection uses 30 random samples from ARC-Easy, and performance is best with in-domain samples; out-of-distribution calibration shows increased variance.