MRPC

anchor

Findings

• IC-1015GPT-J and 10 other LLMs exhibit overthinking: calibrated accuracy given incorrect few-shot demonstrations peaks at a critical layer then declines, and ablating 5 false induction heads in late layers reduces the accuracy gap by 38.9% on average [eval]
• IC-1088ICL predictions in LLaMA, LLaMA-2, and Falcon models depend on in-context label information and can learn truly novel label relationships [eval]
• IC-1089ICL in LLaMA, LLaMA-2, and Falcon models cannot fully overcome pre-training label preferences when in-context labels are flipped [eval]
• IC-1431BERT-base fine-tuning has negligible distribution-wise variance (0.21%) while BERT-large fine-tuning has substantial distribution-wise variance (2.08%) on MRPC [eval]