IC-391Llama-3 and Qwen-1.5 models exhibit position bias in LM-as-a-judge, retrieval-augmented QA, and math reasoning, with larger models showing less bias
Ziqi Wang, Hanlin Zhang, Xiner Li, Kuan-Hao Huang, Chi Han, Shuiwang Ji, Sham M. Kakade, Hao Peng, Heng Ji
The paper measures how swapping the positions of candidate responses, retrieved documents, or math conditions changes model outputs across nine released LLMs. On RewardBench, position bias (the fraction of items where the model changes its answer after swapping) reaches 48.0% for Qwen-1.5-4B-Chat on the chat-hard subset, while Llama-3-70B-Instruct averages only 10.2%. In retrieval-augmented QA, Llama-3-70B-Instruct performs best when the gold document is at the beginning or end of 10 or 20 documents. In math reasoning on R-GSM, Qwen-1.5-7B-Chat and Qwen-1.5-110B-Chat both show position-dependent accuracy. The paper attributes the bias to causal attention and rotary position encoding.
Evidence
correlational
Key metric
Position bias up to 48.0% (Qwen-1.5-4B-Chat, chat-hard); Llama-3-8B-Instruct avg 17.7%, Llama-3-70B-Instruct avg 10.2%; Llama-3-8B vanilla (gt at a) 67.5 vs (gt at b) 66.3; Llama-3-70B vanilla (gt at a) 78.0 vs (gt at b) 76.5; RAG gap +1.2/-2.0 for 10/20 docs; math reasoning improvement 12.6% (7B) and 5.3% (110B) with PINE
Caveat
The paper notes Qwen-1.5-72B-Chat is an exception where PINE does not help, suspecting the model is not well-trained. The RAG experiment uses only Llama-3-70B-Instruct. Temperature is set to 0 to avoid variance.