IC-204GPT-4o performs worse with explicit chain-of-thought prompting than with the original prompt on OpenRCA tasks

Junjielong Xu, Qinan Zhang, Zhiqing Zhong, Shilin He, Chaoyun Zhang, Qingwei Lin, Dan Pei, Pinjia He, Dongmei Zhang, Qi Zhang

SourceOpenRCA: Can Large Language Models Locate the Root Cause of Software Failures?

A repetitive experiment compared GPT-4o's accuracy under the original prompt versus a CoT prompt that explicitly requests step-by-step analysis of telemetry. Across three trials, the median accuracy with CoT was lower in both sampling settings: 2.99% versus 3.58% for balanced sampling, and 5.37% versus 6.27% for oracle sampling. Manual review of a case study showed that CoT led GPT-4o to focus on the most superficially anomalous component (highest latency) rather than following the diagnostic guidance to trace the failure propagation chain to the originating component.

Evidence

correlational

Key metric

GPT-4o median accuracy: balanced 3.58% (original) vs 2.99% (CoT); oracle 6.27% (original) vs 5.37% (CoT)

Caveat

Only GPT-4o was tested in this comparison. The absolute accuracy numbers are very low, so the difference of 0.59 percentage points (balanced) may be within noise despite the median being reported over three trials.

Model

GPT-4o

Concepts

Failure mode

Methods

Chain-of-Thought prompting / Chain-of-Thought (CoT) prompting / CoT prompting / Few-shot Chain of Thought / Few-shot CoT / Chain-of-Thought (CoT-S) / CoT-bag prompting / Few-shot CoT prompting / Rationale prompting / Zero-shot CoT prompting / Wei et al. (2022) chain-of-thought prompting [compared-to]

Related work

Chain-of-Thought prompting / Chain-of-Thought (CoT) prompting / CoT prompting / Few-shot Chain of Thought / Few-shot CoT / Chain-of-Thought (CoT-S) / CoT-bag prompting / Few-shot CoT prompting / Rationale prompting / Zero-shot CoT prompting / Wei et al. (2022) chain-of-thought prompting [builds-on]

Related findings

IC-202, IC-203

Extraction

automatic-extraction