HumanEval

anchor

Findings

• IC-1330All 20 evaluated LLMs improve in multi-turn task-solving with additional tool-use turns and GPT-4-simulated language feedback [eval]
• IC-1413GPT-4 achieves near-saturation on Python code synthesis (86.6% pass@1) but scores significantly lower on code repair (47.8% avg) and code explanation (52.1% avg) across six languages [builds-on]
• IC-1599Self-repair at equivalent compute budget provides only modest and inconsistent gains over i.i.d. sampling for CodeLlama-13B-Instruct, GPT-3.5, and GPT-4 on HumanEval and APPS [eval]
• IC-1600Replacing a model's self-generated feedback with a stronger model's feedback consistently improves self-repair beyond both the i.i.d. baseline and the self-repair baseline [eval]
• IC-388CS-Bench scores correlate strongly (p > 0.9) with math and code benchmark scores across 12 models [eval]
• IC-467Llama-3.1-405B's standard speculative decoding verification rejects correct continuations from GPT-4o, Llama-3.1-8B, and human text, accepting only roughly two tokens before the first rejection for GPT-4o [eval]
• IC-811Alpaca's 52k instruction-tuning data is predominantly low-quality (only 17.75% score ≥ 4.5 on accuracy), yet the full 52k data still yields higher MMLU scores than the filtered 9k subset for both 7b and 13b variants [eval]
• IC-879Code Llama outperforms Llama-2 on coding (HumanEval) and mathematical (GSM8K) reasoning at both 7B and 13B scales [eval]