IC-100GPT-4, GPT-3.5, Claude 3, Llama 3 8B, and Llama 3 70B can be prompted to approximately calibrate their accuracy to specific target percentages on MMLU
Teun van der Weij, Felix Hofstätter, Oliver Jaffe, Samuel F. Brown, Francis Rhys Ward
When prompted to answer exactly p% of MMLU questions correctly (for p in 0, 10, 20, ..., 100), all frontier models can approximately hit the target accuracy. Most models tend to overshoot the target but are clearly capable of modulating their performance. Llama 3 8B was significantly better calibrated than Llama 3 70B. The evaluation uses a subset of MMLU (MMLU-easy, 5,854 samples) that all models can answer correctly, so the full probability range is accessible.
Evidence
correlational
Key metric
All frontier models can accurately approximate arbitrary targets; most tend to overshoot. Llama 3 8B significantly better calibrated than Llama 3 70B.
Caveat
The authors note that performance can likely be further improved with greater engineering effort, such as extended prompt engineering and more extensive hyperparameter sweeps. Evaluation is limited to multiple-choice question answering.