Six LLMs with at least 128k context windows were evaluated on 335 real-world software failure cases requiring identification of root cause elements (component, time, reason) from heterogeneous telemetry. The best model, Claude 3.5 Sonnet, achieved only 11.34% accuracy using the RCA-Agent framework. Performance drops sharply as the number of required root cause elements increases: accuracy falls by at least half going from one to two elements, and all models score 0% on three-element queries. Models also perform better on the simpler telecom system than on the more complex bank and market systems. Proprietary models consistently outperform open-source models across all evaluation methods.
Evidence
correlational
Key metric
Claude 3.5 Sonnet: 11.34% correct (RCA-Agent), 5.37% (oracle), 3.88% (balanced); GPT-4o: 8.96% (RCA-Agent); Gemini 1.5 Pro: 2.69% (RCA-Agent); Llama 3.1 Instruct: 3.28% (RCA-Agent); all models 0.00% on hard (3-element) queries
Caveat
Due to budget constraints, only Llama 3.1 was evaluated on RCA-Agent among open-source models. The 70B version of Llama 3.1 was used rather than 405B because Together.ai does not support 128k context for the 405B version.