On the 2-hop COFCA dataset, the paper categorises every example by whether each sub-question and the final question are answered correctly or incorrectly. GPT-4 achieves a correct final answer via the correct reasoning chain (C-C-C) in only 36.3% of cases, while GPT-3.5 achieves it in only 13.3%. A large proportion of correct final answers come through at least one incorrect sub-answer (e.g., GPT-3.5 has 17.7% incorrect reasoning chains), and total failure (all wrong) accounts for 69% of GPT-3.5's cases. The joint F1-RC and EM-RC scores in Table 7 confirm that reasoning chain quality degrades sharply with hop count, with Bing Chat dropping from joint F1-RC 0.9 (2-hop) to 4.2 (3-hop) to 4.7 (4-hop).
The reasoning chain evaluation is performed only on the 2-hop dataset for the full categorical breakdown. The joint performance metric is sensitive to the first sub-question due to sequential dependency.