ORION

Findings

• IC-372Language models universally decompose retrieval tasks into request processing in middle layers and entity retrieval in late layers at the last token position [eval]
• IC-373In Pythia-2.8B, the specific attention heads and MLPs implementing retrieval depend on superficial input features, and request-patching preserves the natural retrieval mechanism [eval]
• IC-374Pythia models are vulnerable to prompt injection via distractor text, and request-patching from a single trusted input restores most of their accuracy [eval]