IC-097Chain-of-thought prompting outperforms few-shot direct QA for Llama 3 models above 8B parameters, while few-shot is best below 3B

Ricardo Dominguez-Olmedo, Vedant Nanda, Rediet Abebe, Stefan Bechtold, Christoph Engel, Jens Frankenreiter, Krishna P. Gummadi, Moritz Hardt, Michael Livermore

SourceLawma: The Power of Specialization for Legal Annotation

The paper compares three prompting strategies (zero-shot direct QA, 15-shot direct QA, and zero-shot chain-of-thought) across the Llama 3 instruct family. For models below 3B parameters, few-shot direct QA performs best. For models above 8B parameters, chain-of-thought is superior. The largest model evaluated with few-shot, Llama 3.3 70B Instruct, does not benefit from including in-context examples at all. CoT yields modest average improvements of 2-3 accuracy points for both the 8B and 70B models.

Evidence

correlational

Key metric

llama 3 8b instruct 0.503, llama 3 8b instruct + cot 0.529, llama 3 70b instruct 0.623, llama 3 70b instruct + cot 0.640

Caveat

CoT evaluation was limited to Llama 3 8B and 70B Instruct due to compute cost (over 500 H100 GPU hours). Few-shot evaluation used 15 examples, constrained by the 128k context window and 8k token opinions.

Model

Llama-3.2-3B Llama 3.2 1B Instruct, Llama 3.2 3B Instruct, Llama 3.1 8B Instruct

Concepts

Scale-dependent behaviour

Datasets

CaseLawQA [eval]

Methods

Chain-of-Thought prompting / Chain-of-Thought (CoT) prompting / CoT prompting / Few-shot Chain of Thought / Few-shot CoT / Chain-of-Thought (CoT-S) / CoT-bag prompting / Few-shot CoT prompting / Rationale prompting / Zero-shot CoT prompting / Wei et al. (2022) chain-of-thought prompting [primary]

Related findings

IC-096, IC-098

Extraction

automatic-extraction