IC-088OLMo 7B Instruct and Qwen 2.5 1.5B Instruct exhibit a two-stage process for unusual answer symbols, initially assigning non-negligible probability to expected symbols (a/b/c/d) before switching to the actual prompt symbols at a specific later layer
Sarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannaneh Hajishirzi, Ashish Sabharwal
When prompted with unusual answer choice symbols such as q/z/r/x or o/e/b/p, OLMo 7B Instruct and Qwen 2.5 1.5B Instruct first assign non-negligible probability to the more common/expected symbols a, b, c, d even though these are not in the prompt. They then make an abrupt switch to the correct symbols at a specific later layer: layer 29 for OLMo and layer 23 for Qwen. Activation patching confirms this: patching from q/z/r/x prompts only decisively promotes the new symbol from layer 29, noticeably later than layer 24 in the standard a/b/c/d experiments. For the 1/2/3/4 prompt format, scores assigned to a/b/c/d remain negligible, suggesting the two-stage process is specific to letter symbols.
Evidence
interventional
Key metric
OLMo switches to correct symbols at layer 29; Qwen at layer 23; standard a/b/c/d encoding at layer 24
Caveat
The paper notes this could explain why some models struggle with out-of-distribution formats, but does not quantify the performance drop attributable to this two-stage process