SourceSOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents
On the 20 most challenging SOTOPIA tasks (SOTOPIA-HARD), GPT-4 scores 4.85 on goal completion versus 5.95 for humans interacting with GPT-4 and 6.15 for human-human interactions (p<0.05). GPT-4 produces 45.5 words per turn versus 16.8 for humans, indicating less efficient communication. Qualitatively, GPT-4 always rephrases the other agent's utterance before answering (active listening from RLHF training), starts bargaining at its exact target price rather than strategically, and tends to propose compromised solutions rather than persisting in its stated goals.