IC-314Llama-3-70b as an LLM-based evaluator (Llama-3-eval) produces agent rankings consistent with GPT-4-based G-Eval on InsightBench
Gaurav Sahu, Abhay Puri, Juan A. Rodriguez, Amirhossein Abaskohi, Mohammad Chegini, Alexandre Drouin, Perouz Taslakian, Valentina Zantedeschi, Alexandre Lacoste, David Vazquez, Nicolas Chapados, Christopher Pal, Sai Rajeswar, Issam H. Laradji
The paper compares G-Eval (using GPT-4) and Llama-3-eval (using Llama-3-70b) as evaluators of agent-generated insights on the first 10 InsightBench datasets. Both methods produce scores in similar ranges and rank the agents in the same order: GPT-4o backbone highest, then GPT-4-turbo, GPT-3.5-turbo, and Llama-3-70b. The authors conclude Llama-3-70b is a feasible open-source alternative to GPT-4 for evaluation, avoiding API costs and periodic model updates.
Evidence
correlational
Key metric
G-Eval vs Llama-3-eval (soft scores): GPT-4o 0.53±0.01 vs 0.57±0.02; GPT-4-turbo 0.52±0.02 vs 0.55±0.02; GPT-3.5-turbo 0.46±0.01 vs 0.50±0.01; Llama-3-70b 0.45±0.01 vs 0.50±0.02
Caveat
Comparison is on only the first 10 datasets of InsightBench, not the full 100. The two evaluators use different prompt templates (G-Eval vs the custom Llama-3-eval prompt).