The paper evaluates pretrained GPT-2 Large on a syllogism task where the subject term is replaced by a randomly initialized embedding, requiring the model to rely purely on contextual structure rather than token identity. The model shows persistently low accuracy on this unseen-token condition, described as 'even worse than on the pretrained pythia 1.4b.' This demonstrates that a widely used released language model lacks the ability to perform structural ICL on novel tokens in a logical reasoning setting.
Evidence
observational
Caveat
The evaluation is on a single simple syllogism task with a-z tokens; the paper provides no specific accuracy number for the pretrained model, only a qualitative comparison to Pythia-1.4B.