IC-897GPT-4 underperforms codex on Spider text-to-SQL with few-shot prompting, attributed to its zero-shot tuning

Xinyun Chen, Maxwell Lin, Nathanael Schärli, Denny Zhou

SourceTeaching Large Language Models to Self-Debug

The paper evaluates GPT-4, codex, GPT-3.5-turbo, and Starcoder on the Spider development set for text-to-SQL generation using few-shot prompting. GPT-4 achieves a baseline accuracy of 73.2, notably lower than codex at 77.5, and this gap persists with code explanation feedback (73.6 vs 80.8). The authors attribute this to GPT-4 being tuned for zero-shot prompting, making it less effective at following few-shot exemplars. They further note that both GPT-3.5 and GPT-4 show more than 10% worse zero-shot results compared to their few-shot counterparts on this task.

Evidence
correlational
Key metric
Spider dev: GPT-4 baseline 73.2, +expl. 73.6; codex baseline 77.5, +expl. 80.8
Caveat
The authors hypothesize the cause (zero-shot tuning) but do not directly test this hypothesis with a controlled experiment.
Model
GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, code-davinci-002
Datasets
Spider [eval]
Methods
Few-shot prompting / 2-shot prompting / Few-shot ICL / Few-shot prompting for base models [supporting]
Related work
Coder-Reviewer [compared-to]
Related findings
IC-898
Extraction
automatic-extraction