The paper evaluates 10 released LLMs on the MTOb benchmark, which requires translating between English and Kalamang using a 573-page grammar book and a bilingual word list as in-context reference materials. The best-performing model, Claude 2 with word list, parallel sentences, and a ~100k-token grammar book chunk in context, achieves 44.7 CHRF on KGV→ENG and 45.8 CHRF on ENG→KGV. A human who learned Kalamang from the same reference materials scores 51.6 and 57.0 CHRF respectively, a gap of +6.9 and +11.2 CHRF. Model outputs are described as erratic: some are near-perfect while others make nonsensical errors, in contrast to the human's consistently grammatical translations.
Evidence
correlational
Key metric
44.7 chrf (kgv→eng) and 45.8 chrf (eng→kgv) for claude 2 (w+s+gl), vs. 51.6 and 57.0 chrf human baseline
Caveat
The test set is drawn from the same field documentation as the training materials, so it may share vocabulary and structures with the reference book. Kalamang is a single language; generalisation to other typologically diverse languages is untested. The human baseline is a single individual with linguistics training.