IC-372Language models universally decompose retrieval tasks into request processing in middle layers and entity retrieval in late layers at the last token position
Using residual stream patching (a single-input interchange intervention on the residual stream at the last token position), the authors show that across 18 open-source LMs from 125M to 70B parameters and 6 task domains, patching the residual stream at middle layers causes the model to output the request from one input applied to the context of another (request-patching). This reveals a modular decomposition: early layers are undisturbed, middle layers encode the request, and late layers perform the retrieval. The phenomenon is robust across model families, task types, and prompt templates, with 98 of 106 model-task pairs showing at least 70% normalized probability of the cross-combined label token.
Evidence
interventional
Key metric
98 out of 106 pairs of tasks and models demonstrate request-patching with at least 70% normalized probability of r1(c2); l2 (peak request-patching layer) in middle layers; for Llama 2 70B, l2 concentrated in layers 39-43 across all tasks; normalized IIA > 85% in 91 out of 106 settings
Caveat
The phenomenon is absent in the abstract induction task on large models, hypothesised to be due to increased model width and task simplicity. The 80% threshold for defining l1 and l3 is arbitrary.