The paper evaluates ten instruction-tuned models on Skill-Mix(k) for k=2 through k=6, where the model must generate text illustrating k randomly selected skills in a given topic. Performance drops sharply as k increases: most models saturate by k=3 under GPT-4 grading. Within the Llama-2 family, which shares training data and methodology, the saturation point rises with size: Llama-2-7b-chat saturates at k=2, while Llama-2-13b-chat and Llama-2-70b-chat saturate at k=3. GPT-4 is the only model that maintains reasonable performance through k=5 and k=6.
Evidence
correlational
Key metric
Ratio of full marks (GPT-4 grading): Llama-2-7b-chat k=2 .05, k=3 .00; Llama-2-13b-chat k=2 .22, k=3 .05; Llama-2-70b-chat k=2 .28, k=3 .02; GPT-4 k=2 .94, k=3 .68, k=4 .52, k=5 .36, k=6 .29
Caveat
Falcon-180b-chat was evaluated on only 30 combinations with GPT-4 grading due to computation limits, compared to 100 for other models.