Linear Relational Embedding

anchor

Findings

• IC-1553GPT-J, GPT-2-XL, and Llama-13B decode approximately 48% of tested relations via a linear transformation on the subject representation, and this structure causally influences predictions [primary]
• IC-1554LRE faithfulness in GPT-J is concentrated in intermediate layers and drops sharply in later layers, consistent with a mode switch from relational encoding to next-token prediction [primary]