Verbalized Confidence / Verbal confidence elicitation

anchor

Findings

• IC-1264LLMs are overconfident when verbalizing confidence, with values concentrated in 80–100% and multiples of 5, yielding high ECE across all five tested models [primary]
• IC-1265Calibration and failure prediction improve as model capability scales from GPT-3 to GPT-4, but remain far from ideal [primary]
• IC-1266For GPT-3, white-box token-probability methods outperform black-box verbalized confidence in uncertainty estimation, but the gap is narrow (0.522–0.605 AUROC) and both remain near random [compared-to]
• IC-131GPT-4 Turbo, GPT-3.5 Turbo, Llama3-8B, Qwen-7B, and iFlytekSpark-13B over-rely on the strong reminder 'the answer is' in prompts as a shortcut, with accuracy dropping sharply when the cue is a random answer rather than the ground truth [supporting]