IC-985LLaMA-2-7B plateaus in ICL accuracy and fails to override semantic priors when in-context labels are flipped on a simple happy/sad classification task

Ivan Lee, Nan Jiang, Taylor Berg-Kirkpatrick

SourceIs attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability

The paper evaluates the released LLaMA-2-7B checkpoint (used as-is, no fine-tuning) on a simple natural-language ICL task: given in-context sentence pairs linking a character's action to an emotion label (happy/sad), the model must predict the correct label for a new sentence. In the standard setting, accuracy improves with more in-context examples but quickly plateaus, a pattern shared by all 13 architectures tested. In the flipped-label setting, where 'happy' and 'sad' are swapped, LLaMA-2-7B fails to override its pretraining semantic priors and cannot learn the inverted mapping from in-context examples alone, joining the majority of architectures that fail this condition.

Evidence

correlational

Caveat

The task is a very simple binary classification on 200 short sentence pairs, far from the model's pretraining distribution; specific accuracy values are reported only in Figure 4 and not in the text. The model is evaluated on a task type (few-shot ICL on short sentences) that differs substantially from its pretraining objective.

Model

Llama 2 / Llama 2 base

Concepts

Failure mode

Extraction

automatic-extraction