IC-467Llama-3.1-405B's standard speculative decoding verification rejects correct continuations from GPT-4o, Llama-3.1-8B, and human text, accepting only roughly two tokens before the first rejection for GPT-4o
Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, Jonas K Kohler
The paper measures how many candidate tokens Llama-3.1-405B accepts under standard speculative decoding verification when the draft is a high-quality source. Using GPT-4o as the draft on MT-Bench, GSM8K, and HumanEval, the target accepts only roughly two tokens before the first rejection, despite the answers being correct. The same pattern holds for Llama-3.1-8B as draft (correct solutions rejected after a few tokens) and for human-written Wikipedia summaries. Reversing the roles (405B drafting for 8B) also yields low acceptance. The authors conclude that acceptance rates do not improve with the quality of the responses, because the verification criterion measures alignment with the target's own distribution rather than contextual correctness.
Evidence
correlational
Key metric
"accepting only roughly two before encountering the first rejection" (GPT-4o as draft for Llama-3.1-405B, greedy decoding, MT-Bench/GSM8K/HumanEval)
Caveat
The GPT-4o experiment uses greedy decoding and only checks the first rejection point, since full speculative decoding is not possible with closed-source models. The human-text experiment uses a subset of Wikipedia summaries with a specific prompting format.