SourceWalk the Talk? Measuring the Faithfulness of Large Language Model Explanations
On 30 MedQA type-2 questions (hypothetical patient visits), the same causal concept faithfulness method is applied using removal-based counterfactuals. All three models receive low dataset-level faithfulness scores: GPT-3.5 f(x)=0.50, GPT-4o f(x)=0.34, Claude-3.5-Sonnet f(x)=0.30. The dominant unfaithfulness pattern is that explanations omit concepts with the largest causal effects (e.g., the patient's mental status upon arrival has CE=0.32 for Claude but EE=0) while frequently citing concepts with much smaller effects (e.g., vital signs with CE=0.07 but EE=1.00 for Claude). This means a user reading the explanation would incorrectly conclude that vital signs drive the diagnosis while mental status is irrelevant.