SourceSKILL-MIX: a Flexible and Expandable Family of Evaluations for AI Models
Using upper bounds on skill frequency (p_s ≤ 0.0144), topic frequency (p_t ≤ 0.0022), and corpus size (l ≤ 5×10^10 sentences) estimated from the RedPajama dataset, the paper shows that the expected number of training text pieces containing any given k-skill and topic combination is at most 0.07 for k=5 and 0.001 for k=6. GPT-4's observed ratio of full marks (α5 ≈ 0.12, α6 ≈ 0.08 after filtering common skills) exceeds the threshold (3/2)·p^k_s·p_t·l, implying more than one-third of its correct answers involve combinations never jointly seen in training. No other model shows similar evidence.