When processing in-context graph traces, Llama-3.1-8B's earlier layers (e.g., layer 6) show token representations organized by pretrained semantic relationships, while deeper layers (e.g., layer 26) show representations organized by the in-context graph geometry. In the ring graph, semantically related concepts (apple, orange) are closer together in layer 6 but become essentially antipodal by layer 26, matching the graph's structure. Layer-by-layer PCA projections (Figs. 9, 10) show the grid and hexagonal structures becoming progressively clearer from layer 0 through layer 30.
Evidence
correlational
Key metric
In the ring graph, concepts apple and orange are closer to each other in layer 6 but become essentially antipodal around layer 26, as dictated by the graph.
Caveat
The depth-dependent observation is based on visual inspection of 2D PCA projections, which the authors acknowledge can be difficult to interpret or misleading.