SourceRevisiting In-context Learning Inference Circuit in Large Language Models
The paper measures hidden states of forerunner tokens (the token immediately before the label, e.g. ':') across all transformer layers in Llama 3 70B and three other models. Using mutual nearest-neighbor kernel alignment against BGE M3 sentence embeddings and centroid classification, it shows that the forerunner token produces the best input text encoding, emerging in layers 0-28 and maintaining high alignment through the final layer. The encoding is linearly separable (centroid classifier achieves high accuracy) and is enhanced by increasing the number of in-context demonstrations. Perplexed inputs are encoded worse in zero-shot but this effect disappears with demonstrations, indicating a hybrid in-weight and in-context encoding process.