Comparing vanilla verbalized confidence across GPT-3 (175B), GPT-3.5, and GPT-4 reveals a consistent trend: as model capability increases, ECE decreases and AUROC increases. The paper reports an approximate 22.2% improvement in AUROC from GPT-3 (avg 51.3) to GPT-4 (avg 62.7). However, even GPT-4's AUROC remains close to the 50% random-guess threshold, and its ECE of 18.0 (×10²) still indicates substantial miscalibration. The improvement is monotonic across the GPT family but does not reach satisfactory performance.
Evidence
correlational
Key metric
Approximate 22.2% improvement in AUROC from GPT-3 to GPT-4; average ECE (×10²): GPT-3 52.0, GPT-3.5 37.7, GPT-4 18.0; average AUROC: GPT-3 51.3, GPT-3.5 55.1, GPT-4 62.7
Caveat
The comparison spans different model families (GPT-3/3.5/4, Vicuna, Llama 2) rather than a single family's size variants, so the scaling trend conflates architecture changes with scale. The paper itself notes results remain 'far from ideal performance.'