The paper benchmarks nine released models on a new 1,073,146-question multimodal STEM dataset spanning science, technology, engineering, and math at K-12 grade levels. The best-performing model, CLIP-RN50x64, achieves 54.9% average accuracy, while GPT-3.5-turbo reaches 49.0%. Seven university students attain 83.0% accuracy. The average exam score for elementary grades (1-6) is 40.8, which is 54.7% lower than the human reference of 90. Models perform well on object-identification skills but fail on skills requiring abstract reasoning, comparison, and complex spatial understanding. Math is the hardest subject for all models, with only marginal improvements over random chance.
Evidence
correlational
Key metric
CLIP-RN50x64 54.9% avg accuracy, GPT-3.5-turbo 49.0%, GPT-3 46.7%, UNITER 45.9%, ViLBERT 39.5%, 12-in-1 38.3%, ViRTex 37.1%, UnifiedQA-base 41.7%, GloVe 37.6%, random 36.9%, human 83.0%; exam score 40.8 vs human 90 (54.7% lower); 2.5% third-grade skills mastered
Caveat
The technology subset has no grade-level information; the human comparison uses 7 university students on 80 sampled questions for accuracy and IXL SmartScore for exam scores.