SourcePandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
The paper reports an observed inherent bias in GPT-3.5 where its evaluation of which response is better depends on the order in which the two responses are presented in the prompt. Even with carefully designed prompts, GPT-3.5's judgment conflicts when the response order is swapped. The authors mitigate this by filtering out inconsistent training samples and by swapping the order at inference time, revising conflicting results to 'tie'. No specific magnitude or proportion of affected samples is reported.