SourceSOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents
In SOTOPIA's interactive multi-turn social interactions, Llama-2-70B-Chat scores lower than GPT-3.5 on every dimension (goal 5.38 vs 6.45, believability 8.10 vs 9.15, knowledge 3.11 vs 3.40, relationship 0.91 vs 1.23, financial 0.40 vs 0.46). This inverts the ranking seen on static language understanding benchmarks where Llama-2-70B-Chat is on par or better than GPT-3.5. The authors hypothesize this is because Llama-2-70B-Chat is less heavily trained on human feedback/user interaction data. Qualitative inspection shows Llama-2-70B-Chat struggles to maintain persona, move conversations forward, and respond actively to the other agent.