IC-121Gemini 1.0 Pro, when prompted as a zero-shot chain-of-thought judge with majority voting, underperforms fine-tuned smaller Gemma models as verifiers on GSM8K

Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, Rishabh Agarwal

SourceGenerative Verifiers: Reward Modeling as Next-Token Prediction

The paper uses Gemini 1.0 Pro as an LLM-as-a-judge, prompting it with a step-by-step verification template and averaging over 32 sampled CoT rationales via majority voting. Despite being a more capable model than the Gemma variants used as verifiers, Gemini 1.0 Pro's zero-shot verification accuracy is consistently lower than that of fine-tuned Gemma-2B, 7B, and 9B GenRM-COT verifiers on GSM8K. This holds across all three Gemma model scales and across all numbers of majority votes from 1 to 32. The authors note that the finetuned verifier outperforms the judge 'which also utilizes the same cot approach and number of majority votes, but prompts a more capable gemini 1.0 pro model than gemma models which we finetune as verifiers.'

Evidence

correlational

Key metric

73% pass@1 (Gemini 1.0 Pro generator on GSM8K); GenRM-COT (Gemma-2 9B) best-of-16: 93.4%; LLM-as-a-judge (Gemini 1.0 Pro) underperforms GenRM-COT at all model scales (2B, 7B, 9B) and all vote counts (1-32) per Figure 7

Caveat

The comparison is on a single task (GSM8K) with a specific prompting template; the LLM-as-a-judge uses the same model (Gemini 1.0 Pro) that generated the solutions being verified, which may introduce self-bias

Model

Gemini 1.0 Pro

Concepts

Scale-dependent behaviour

Datasets

GSM8K [eval]

Methods

LLM-as-a-Judge / GPT-4 as judge / GPT-4o as LLM judge [compared-to]

Extraction

automatic-extraction