The paper measures expected calibration error (ECE) for the confidence scores each model assigns to its internal answer (internalconf) versus its context-based answer (contextconf) across six datasets. For Llama-3-8B, internal ECE ranges from 13.6 to 23.6 while context ECE ranges from 32.8 to 43.1. For GPT-4o mini, internal ECE is 7.7–25.9 versus context ECE 30.4–45.9. For GPT-4o, internal ECE is 6.5–27.6 versus context ECE 25.0–39.9. The gap is consistent across all three models and all datasets, indicating a systematic asymmetry in the models' ability to produce well-calibrated confidence for external information.
Evidence
correlational
Key metric
ECE: Llama-3-8B internal 13.6–23.6 vs context 32.8–43.1; GPT-4o mini internal 7.7–25.9 vs context 30.4–45.9; GPT-4o internal 6.5–27.6 vs context 25.0–39.9 (across TriviaQA, PopQA, NaturalQA, FreshQA, RedditQA, ClashEval)
Caveat
Calibration is measured on the confidence scores elicited by specific prompting procedures; the gap may narrow or widen under different elicitation methods. The paper notes that improving ECE via isotonic regression does not always translate to better situated faithfulness.