Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
ICLR: In-Context Learning of Representations
2025-01-22
· ICLR 2025 Poster ·
anchor
Findings
IC-481
Llama-3.1-8B and four other released LLMs reorganize their internal representations to reflect in-context graph structure in a sudden two-phase transition as context length increases
IC-482
In Llama-3.1-8B, in-context graph structure is absent in early layers dominated by semantic priors and emerges clearly in deeper layers
IC-483
When in-context graph structure conflicts with pretrained semantic priors, Llama-3.1-8B encodes the in-context structure in higher principal components while the semantic prior dominates the first two
IC-484
Re-scaling the first 2-3 principal components of Llama-3.1-8B token representations causally shifts next-token predictions toward the target graph position