IC-483When in-context graph structure conflicts with pretrained semantic priors, Llama-3.1-8B encodes the in-context structure in higher principal components while the semantic prior dominates the first two
Using days of the week (which have a circular semantic representation from pretraining) as nodes on a non-semantic 7-node ring, the model's first two principal components still show the original semantic ring geometry, while the in-context ring structure appears in the third and fourth principal components. The Dirichlet energy minimization occurs specifically in the PCA 3,4 dimensions corresponding to the in-context structure. Despite the semantic prior dominating the leading PCs, the model transitions from making semantic-prior predictions to in-context-structure predictions as context length increases, eventually performing well on the task.
Evidence
correlational
Key metric
Energy minimization occurs in PCA 3,4 (in-context dimensions) rather than PCA 1,2 (semantic dimensions), as shown in Fig. 15. The model transitions from semantic-prior accuracy to in-context accuracy with increasing context (Fig. 14).
Caveat
The in-context structure is present but 'unable to dominate the pretrained structure.' The authors note this is a limitation of the representation reorganization under semantic conflict.