The paper measures whether released language model checkpoints can determine part-of-speech (MultiBERTs) or complete a syllogism (Pythia-1.4B) when the query token is replaced by a randomly initialized embedding, i.e. when no in-weight information is available. In both model families, probe accuracy on these unseen-token evaluations spikes early in training and then decays to chance or below-chance levels by the end of training. This means the ability to generalize to truly novel tokens is an early, transient inductive bias that is overwritten as the model memorizes token identities into its embeddings.
Evidence
observational
Caveat
The MultiBERTs results are on a POS probing task with templatic sentences; the Pythia-1.4B results are on a simple syllogism task. Both are simplified evaluations of structural ICL rather than open-ended generation.