IC-266GPT-4o drops from 96.3% closed-book accuracy to 47.5% when given counterfactual context that contradicts its parametric knowledge, far below the 95% human accuracy on the same items
On the counterfactual context task, a multi-paragraph context is generated that provides fabricated evidence supporting an answer that contradicts well-known facts (e.g., 'wood is magnetic'). In the closed-book setting (no context), nearly half the models exceed 90% accuracy, with GPT-4o at 96.3%. When the counterfactual context is added, performance drops sharply: GPT-4o falls to 47.5%. A human study on a held-out subset shows 95% accuracy, indicating the correct answer is derivable from the context. The gap between model and human performance highlights a faithfulness limitation: models cannot override their parametric knowledge in favor of the provided context.
Evidence
correlational
Key metric
GPT-4o: 96.3% (closed-book) vs 47.5% (counterfactual context); human accuracy 95% on held-out subset; nearly half of models achieve over 90% closed-book
Caveat
The counterfactual contexts were generated by GPT-4o and validated via string matching (68.9% pass rate). The task uses multiple-choice format from ARC-Challenge, which may constrain the type of counterfactuals tested.