When self-debugging is performed without unit test execution, the model must rely solely on self-generated feedback to assess correctness. GPT-3.5-turbo shows no improvement on Transcoder (89.1 to 89.1) and only 0.6% on MBPP (67.6 to 68.2). GPT-4 shows 0.9% on Transcoder (77.3 to 78.2) and 3.6% on MBPP (72.8 to 76.0). The authors explicitly state that both models 'tend to be overconfident in their initial predictions' in this setting, in contrast to codex which still improves by up to 5% without execution.
Evidence
correlational
Key metric
Transcoder without execution: GPT-3.5 89.1→89.1, GPT-4 77.3→78.2; MBPP without execution: GPT-3.5 67.6→68.2, GPT-4 72.8→76.0
Caveat
The overconfidence is observed specifically in the self-debugging prompting context; it is unclear whether it generalizes to other self-correction or self-evaluation settings.