Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
CHRF
anchor
Findings
IC-1508
LLMs with in-context learning translate Kalamang-English at 44.7/45.8 CHRF, falling short of the human baseline of 51.6/57.0 CHRF
[eval]
IC-1509
Kalamang-English translation performance on MTOb increases with model size within the Llama and Llama 2 families, and GPT-4 outperforms Text-davinci-003
[eval]
IC-1510
Without retrieved context, LLMs are unable to translate Kalamang, and among context types, retrieved parallel sentences are most beneficial, followed by word list entries, then grammar book passages
[eval]