IC-165Llama-3-8B performs algorithm selection during in-context learning, selecting the classification criterion with the lowest test error on ambiguous natural language tasks

Qixun Wang, Yifei Wang, Xianghua Ying, Yisen Wang

SourceCan In-context Learning Really Generalize to Out-of-distribution Tasks?

The paper designs an ambiguous sentence classification task where each sentence can be classified by one of three criteria (sentiment, type, or location), with labels mapped to meaningless strings. As context length increases, the model's top-5 accuracy for the true underlying criterion significantly exceeds the accuracy for the other two criteria. This demonstrates that the model performs algorithm selection by preferring the pretraining function that minimizes test error, consistent with the theoretical low-test-error preference proven in section 5.1.

Evidence

correlational

Caveat

The specific model used in section 5.3 is not explicitly named in the text; it is inferred to be Llama-3-8B based on the paper's pattern of using this model for all real-world LLM experiments. The task uses only 128 test examples per context length.

Model

Llama 3 8B

Methods

In-Context Learning / In-context learning prompt [primary]

Related work

Yadlowsky et al. (2023): Pretraining data mixtures enable narrow model selection [compared-to]

Related findings

IC-164

Extraction

automatic-extraction