IC-327Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3, and several other LLMs produce well-calibrated verbal confidence estimates on classification tasks
The paper measures how well each model's self-reported verbal confidence (highest/high/medium/low/lowest) aligns with actual correctness using ECE and AUROC, averaged over 500 examples. Llama-3-8B-Instruct achieves ECE of 0.1264 (SST-2), 0.1376 (AG-News), and 0.0492 (StrategyQA) with AUROC above 0.66 on all three. Mistral-7B-Instruct-v0.3 shows similar calibration. Appendix F.1 extends this to Gemma2, Mistral-Nemo, Qwen2.5, and Llama-3.2-11B, all showing low ECE. GPT-4o shows the lowest ECE (0.0286 on SST-2). The authors conclude that well-calibrated models for confidence elicitation already exist and this capability is likely to improve with scale.
Calibration is measured in a zero-shot classification setting with a specific two-step prompting strategy (k guesses then verbal confidence). The StrategyQA AUROC is notably lower (0.66 for Llama-3) than SST-2 (0.97), suggesting calibration degrades on harder reasoning tasks.