When GPT-3.5-turbo fails on a hard reasoning question under chain-of-thought prompting, re-prompting it with a different set of CoT demonstrations typically yields the same incorrect answer. In contrast, program-of-thought prompting tends to produce a different (or correct) answer. Among cases where CoT fails, 43% are solved by PoT and 57% receive a different wrong answer from PoT. This error-correlation structure is what makes mixing CoT and PoT (MOT) more effective at detecting model uncertainty than either representation alone.
Evidence
correlational
Key metric
43% of CoT-failure cases solved by PoT, 57% different wrong answer from PoT (Appendix D case analysis)
Caveat
The case analysis is illustrative and the 43%/57% split is reported without a formal sample size or confidence interval; the pattern is demonstrated on a subset of hard questions.