Arena-Hard-Auto

anchor

Findings

• IC-209LLM judges (GPT-3.5-turbo-1106, GPT-4o-mini, GPT-4o, Claude-3-5-sonnet) implicitly prioritize style over factuality and safety when scoring pairwise preferences [eval]
• IC-210GPT-4o-mini-2024-07-18 does not exhibit authority bias when used as a judge: appending fabricated references to model responses decreases rather than increases its score [eval]