IC-080GPT-4's performance is largely insensitive to the content of feedback; simple re-prompting with a sound verifier (sampling) matches or exceeds detailed critique
Kaya Stechly, Karthik Valmeekam, Subbarao Kambhampati
The paper ablates the feedback content in the iterative loop, comparing binary feedback ('wrong'), first-error feedback, and all-error feedback from a sound verifier. The differences between these conditions are small, and in two domains (graph coloring, mystery blocksworld) more detailed feedback actually decreases performance. Removing critique entirely and simply re-prompting the same question until the sound verifier accepts an answer (sampling) achieves comparable gains. A self-consistency baseline (picking the most common answer from 15 samples) shows no improvement over standard prompting, confirming that the gain comes from the sound verifier filtering, not from the model learning from feedback.
The sampling setup has quadratically lower token cost than the critique setup because prompts do not grow with iteration count. The authors note that increasing k further can push performance higher, so the comparison at k=15/25 may understate the sampling advantage.