The paper replicates the multi-agent debate method of Du et al. (2023) using three GPT-3.5-turbo-0301 agents over two rounds on the full GSM8K test set. At 6 responses, multi-agent debate scores 83.2 versus 85.3 for self-consistency; at 9 responses, 83.0 versus 88.2. The authors conclude that the improvement attributed to 'debate' is actually due to self-consistency (sampling and voting), not to the critique mechanism.
Only one model (GPT-3.5-turbo-0301) and one dataset (GSM8K) are tested for this comparison. The authors use the exact prompt from Du et al. (2023) with 3 agents and 2 rounds.