All three large LMs evaluated show substantially lower ECE on SCIQ (1000 science exam questions) than on TruthfulQA (817 questions designed to test mimicry of human misconceptions). GPT-4o drops from ECE 0.07 to 0.22, Claude-3.5-Sonnet from 0.06 to 0.25, and Gemini-1.5-Pro from 0.17 to 0.28. Accuracy also drops sharply (e.g., GPT-4o from 0.73 to 0.35), suggesting the harder dataset exposes both lower discriminative ability and worse calibration.
Evidence
correlational
Key metric
GPT-4o: SCIQ ece 0.07 vs TruthfulQA ece 0.22; Claude-3.5-Sonnet: SCIQ ece 0.06 vs TruthfulQA ece 0.25; Gemini-1.5-Pro: SCIQ ece 0.17 vs TruthfulQA ece 0.28
Caveat
The calibration gap may be confounded by the large drop in accuracy on TruthfulQA (e.g., GPT-4o acc 0.73 vs 0.35), since calibration is harder to achieve when the model is less discriminative.