IC-165Llama-3-8B performs algorithm selection during in-context learning, selecting the classification criterion with the lowest test error on ambiguous natural language tasks
The paper designs an ambiguous sentence classification task where each sentence can be classified by one of three criteria (sentiment, type, or location), with labels mapped to meaningless strings. As context length increases, the model's top-5 accuracy for the true underlying criterion significantly exceeds the accuracy for the other two criteria. This demonstrates that the model performs algorithm selection by preferring the pretraining function that minimizes test error, consistent with the theoretical low-test-error preference proven in section 5.1.
Evidence
correlational
Caveat
The specific model used in section 5.3 is not explicitly named in the text; it is inferred to be Llama-3-8B based on the paper's pattern of using this model for all real-world LLM experiments. The task uses only 128 test examples per context length.