IC-1555GPT-J's internal representations contain correct factual knowledge even when the model outputs falsehoods under repetition or instruction distraction prompts
The authors apply the attribute lens (a relation-specific linear probe derived from the LRE) to GPT-J's hidden states on 11,891 repetition-distracted and 11,891 instruction-distracted prompts designed to bait the model into outputting wrong answers. While GPT-J almost never outputs the correct fact (r@1 = 0.02 for repetition-distracted, 0.03 for instruction-distracted), the attribute lens recovers the correct answer in the top-3 predictions 71% and 78% of the time respectively, revealing that the correct knowledge is present in the internal representation but does not reach the output under these adversarial conditions.
Evidence
correlational
Key metric
r@1: model 0.02 vs attribute lens 0.54 (repetition-distracted); model 0.03 vs attribute lens 0.63 (instruction-distracted); r@3: 0.41 vs 0.71 and 0.25 vs 0.78; 11,891 prompts per condition
Caveat
The attribute lens is a linear probe derived from the LRE, so it can only reveal knowledge that is linearly encoded; the finding is limited to relations for which a good LRE exists; first-token evaluation criterion applies