IC-1261LLaMA-2 attention to constraint tokens correlates with factual correctness, and a linear probe on these attention weights predicts factual errors comparably to model confidence
Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, Besmira Nushi
The paper measures the attention contribution from constraint tokens to the final generation token across all layers and heads in LLaMA-2 7B, 13B, and 70B. When the model produces a factually correct completion, the norm of attention to constraint tokens is larger; when it fails, attention is weaker. A logistic regression probe (SAT Probe) trained on these attention weights achieves AUROC comparable to the model's own confidence for predicting factual errors across 10 datasets with over 40,000 prompts. The combined predictor (attention + confidence) outperforms either alone in most settings. The signal is also available for individual constraints in multi-constraint queries.
Evidence
correlational
Key metric
Spearman's ρ = 0.67 (70B), 0.65 (13B), 0.66 (7B) for predicting entity popularity from attention (p < 10^-8); SAT Probe AUROC 0.95 ± 0.00 vs confidence 0.96 ± 0.00 (7B basketball players); SAT Probe AUROC 0.73 ± 0.00 vs confidence 0.88 ± 0.00 (70B basketball players); combined predictor AUROC 0.91 ± 0.00 (70B basketball players)
Caveat
The paper acknowledges that exact match is a strict criterion that could introduce noise; attention alone does not explain all failures (some attention on constraints where the model still fails); the content of the information in attention patterns remains opaque