SourceThe Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”
In an in-context version of the reversal experiment (Appendix B.6), the authors prompted GPT-3 models with a statement in one order (e.g., 'the description is the name') using 3-shot demonstrations at temperature 0, then asked the model to produce the reverse. All GPT-3 sizes achieved 100% accuracy on name-to-description reversal, and 96.67-100% on description-to-name reversal. This contrasts sharply with the fine-tuning results where the same model sizes fail to reverse after training, indicating the model has the reasoning capacity to reverse but the auto-regressive training process does not produce bidirectional associations.