IC-181Truthfulness in Mistral-7B, Mistral-7B-Instruct, Llama3-8B, and Llama3-8B-Instruct is linearly decodable from internal representations at exact answer tokens, with middle-to-late layers being most informative
Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, Yonatan Belinkov
The paper trains linear probing classifiers on intermediate activations of four LLMs to predict whether a generated answer is correct. Probing the exact answer tokens (the tokens whose modification would change the answer's correctness) yields substantially higher AUC than probing the last generated token, the token before it, or the end of the prompt. For example, on TriviaQA with Mistral-7B-Instruct, the probe at the exact answer token achieves AUC 0.85 versus 0.71 at the last generated token. The pattern is consistent across all four models and ten datasets. Additionally, middle-to-late layers consistently outperform early layers for probing, indicating a depth-dependent organisation of the truthfulness signal.
Evidence
correlational
Key metric
Mistral-7B-Instruct TriviaQA: probe exact 0.85±0.004 vs last generated 0.71±0.006; Winobias: 0.92±0.005 vs 0.82±0.004; Math: 0.92±0.008 vs 0.74±0.008. Llama3-8B-Instruct TriviaQA: 0.83±0.002 vs 0.81±0.005; Winobias: 0.93±0.004 vs 0.86±0.007; Math: 0.95±0.027 vs 0.82±0.016
Caveat
The paper focuses on QA tasks with clear gold labels; open-ended tasks are left for future research. The exact answer token extraction relies on an external instruct LLM or heuristics, which may introduce noise.