SOTOPIA-EVAL

Findings

• IC-1401GPT-4 serves as a proxy for human judgment on SOTOPIA-EVAL, with strong correlations on goal, financial, and relationship dimensions for model outputs [primary]
• IC-1402Llama-2-70B-Chat underperforms GPT-3.5 across all SOTOPIA dimensions in interactive social scenarios, diverging from static benchmark rankings [primary]
• IC-1403All four evaluated LLMs produce negative scores on social rules and secret-keeping dimensions in SOTOPIA interactions [primary]
• IC-1404On SOTOPIA-HARD, GPT-4 achieves significantly lower goal completion than humans and exhibits non-strategic negotiation and excessive compromise behaviors [primary]