MBPP

anchor

Findings

• IC-1231On APPS, released code generation models span pass@1 from 0.20 (GPT-3 175B) to 6.20 (CodeRL), with value-based and policy-based RL methods outperforming supervised baselines [eval]
• IC-1330All 20 evaluated LLMs improve in multi-turn task-solving with additional tool-use turns and GPT-4-simulated language feedback [eval]
• IC-1342Assigning socio-demographic personas to LLMs causes significant reasoning performance degradation across all four models studied, manifesting as both explicit abstentions and implicit reasoning errors [eval]
• IC-148Language models represent semantically equivalent inputs from different data types (languages, code, images, audio) close together in intermediate layers, with the shared space scaffolded by the model's dominant language [eval]
• IC-149Intervening in the shared representation space using the dominant language (English) predictably changes model outputs for other data types, demonstrating the space is causally used rather than a vestigial byproduct [eval]
• IC-388CS-Bench scores correlate strongly (p > 0.9) with math and code benchmark scores across 12 models [eval]
• IC-898GPT-3.5-turbo and GPT-4 are overconfident in their initial code predictions when unit test execution is unavailable [eval]