The paper tests whether models can reverse a forward mapping: given an attribute value, find the entity that has it. On Jane Austen novels, GPT-4 scores 0.8% on 'what sentence comes before this one?' versus 65.9% on the forward direction. On Wikipedia celebrity data, GPT-4 scores 42% inverse versus 99% forward; GPT-3.5 scores 23.9% versus 89.5%. On Chinese poems, GPT-4 scores 7.3% inverse versus 66.5% forward. The paper's synthetic experiments confirm this failure is inherent: no amount of training data, model size, or prompting strategy enables inverse search unless the inverse form is explicitly present in training data. GPT-4o and Llama-3.1-405B show the same failure.
Evidence
correlational
Key metric
GPT-4: 0.8% inverse vs 65.9% forward (Jane Austen); 42% inverse vs 99% forward (WikiBio); 7.3% inverse vs 66.5% forward (Chinese poem). GPT-3.5: 0.5% vs 14.4% (Austen); 23.9% vs 89.5% (WikiBio); 2.1% vs 33.0% (poem). GPT-4o and Llama-3.1-405B: 'still fail' (Figure 5).
Caveat
The paper acknowledges inverse search may be challenging even for humans, but notes the Chinese idiom and poem tasks are 'allegedly simple for many high school graduates in Chinese education.' The GPT-4o and Llama-3.1-405B results are qualitative.