IC-178Self-guided confidence reasoning (SCR) outperforms rule-based confidence reasoning (RCR) for GPT-4o and GPT-4o mini, but RCR outperforms SCR for Llama-3-8B
The paper compares two families of confidence-reasoning strategies across three models. For GPT-4o, explicit SCR achieves overall situated faithfulness of 76.4 versus 74.3 for the best RCR method (internalconf); for GPT-4o mini, 69.5 versus 66.9. For Llama-3-8B, the pattern reverses: explicit SCR scores 53.5 while internalconf scores 58.7. The authors attribute the reversal to Llama-3-8B's weaker reasoning ability, which makes it unable to effectively compare internal and external confidence in free-form text, whereas a simple threshold rule on its internal confidence works better.
Evidence
correlational
Key metric
explicitSCR ov: GPT-4o 76.4, GPT-4o mini 69.5, Llama-3-8B 53.5; internalconf ov: GPT-4o 74.3, GPT-4o mini 66.9, Llama-3-8B 58.7
Caveat
SCR and RCR are prompting/decision strategies applied to the same released model; the comparison reflects the interaction between model capability and strategy, not an intrinsic property of the model alone.