Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations
2025-01-22
· ICLR 2025 Spotlight ·
anchor
Findings
IC-082
GPT-3.5, GPT-4o, Claude-3.5-Sonnet, and Llama-3.1-8B produce explanations on the BBQ social bias task that are systematically unfaithful for identity and behavior concepts while remaining faithful for context concepts, with specific patterns of hiding safety-measure influence and social bias
IC-083
GPT-3.5, GPT-4o, and Claude-3.5-Sonnet produce unfaithful explanations on MedQA medical questions, omitting high-effect clinical concepts such as the patient's mental status while over-referencing low-effect concepts