Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
OctoPack: Instruction Tuning Code Large Language Models
2024-01-16
· ICLR 2024 spotlight ·
anchor
Findings
IC-1413
GPT-4 achieves near-saturation on Python code synthesis (86.6% pass@1) but scores significantly lower on code repair (47.8% avg) and code explanation (52.1% avg) across six languages
IC-1414
Pretrained code models Starcoder and CodeGeex2 score 0.0% on code explanation across all six languages because they generate code instead of natural language
IC-1415
BLOOMZ generalizes instruction-following to programming languages (Go, Rust) absent from its instruction data, scoring above the random baseline