IC-1090ICL in LLaMA, LLaMA-2, and Falcon models preferentially uses in-context label information closer to the query rather than treating all examples equally
The paper tests whether ICL considers all in-context information equally (NH3) by changing the label relationship mid-context in three scenarios: default-then-flipped (d→f), flipped-then-default (f→d), and alternating (f↔d). After observing the same number of examples of both label relationships (2n total), predictions are significantly different across the three setups. The f→d setup biases toward default labels while d→f biases toward flipped labels, indicating the model weights recent examples more heavily. This is distinct from the label-frequency bias of Zhao et al. (2021) because it concerns the position of the label relationship change, not the frequency of individual labels.
Evidence
correlational
Caveat
Results are reported for the largest models (LLaMA-2-70B, LLaMA-65B, Falcon-40B) on tasks where label flipping gave strong performance. The authors note that for LLaMA-2-70B on hate speech, metrics appear similar initially but significant differences emerge at additional changepoints.