SourceRocketEval: Efficient automated LLM evaluation via grading checklist
The paper measures the ratio of disagreement across repeated samplings of binary yes/no judgments on WildBench checklist items for five released LLMs. When asked to judge whether a response satisfies a specific criterion, the models produce inconsistent answers across samplings. Qwen2-1.5B shows a disagreement ratio exceeding 50% across 3 sampling results, while larger models like Llama-3-70B show much lower disagreement. The paper identifies a clear correlation between model size and consistency, with smaller models tending to produce higher inconsistency. This high uncertainty is identified as a primary reason why lightweight LLMs fail as reliable judges in chain-of-thought evaluation settings.