When the RCA-Agent encounters a failed code execution, the model must use the error feedback to revise its code or adjust its reasoning. Manual review showed that GPT-4o and Claude 3.5 effectively utilize execution feedback (empty output, exception tracebacks) to recover, while Gemini 1.5 Pro rarely does. Quantitatively, Gemini's accuracy on queries involving failed executions drops from 2.69% to 0.85% (a 68.4% reduction), compared to 17.9% for Claude and 15.6% for GPT-4o. This indicates that the agent framework imposes an additional error-tolerance requirement that Gemini does not meet.
Evidence
correlational
Key metric
Gemini 1.5 Pro: 0.85% correct on failed-execution queries, 68.4% drop; Claude 3.5: 9.31%, 17.9% drop; GPT-4o: 7.56%, 15.6% drop; Llama 3.1: 3.18%, 18.04% drop
Caveat
The analysis is based on manual review of intermediate agent steps and a subset of queries involving failed executions. The absolute accuracy numbers are very low, so percentage drops may be sensitive to small absolute differences.