In appendix H.3, the authors evaluate GPT-2, Llama-2-7B, Llama-2-13B, and Llama-2-70B directly on conjunction and majority tasks at 5, 7, and 15 dimensions. GPT-2 fails on both tasks even at 5 dimensions, performing near the null classifier. Llama-2-7B achieves non-trivial performance, and accuracy increases gradually through 13B to 70B. This indicates that the ability to implement learning algorithms in-context emerges and strengthens with model scale.
Evidence
correlational
Key metric
GPT-2 fails on both tasks even for 5 dimensions; gradual increase in performance as Llama-2 size increases from 7B to 13B to 70B
Caveat
Only three Llama-2 sizes are tested; the relationship between scale and performance is observed, not controlled. GPT-2 is a different architecture family, so the comparison is not purely within one family.