SourceCompetition Dynamics Shape Algorithmic Phases of In-Context Learning
The authors feed sequences generated from a finite Markov chain into Llama-3.1 at three scales (8B, 70B, 405B) and measure the KL divergence between the model's predicted next-state distribution and the ground-truth transition matrix. Without being told the sequence is Markovian, all three models achieve a KL between the unigram-inference baseline and the bigram-inference baseline, showing they extract some transition structure from context. Performance improves with scale: 405B achieves the lowest KL, followed by 70B, then 8B.