IC-351GPT-4-turbo-2024-04-09, Qwen2-72b-instruct, and Llama-3.1-70b-instruct show distinct performance profiles across ultra-long, 32k, and 4k context benchmarks
The paper evaluates several released long-context LLMs on three benchmark suites covering different context lengths. On ultra-long (>100k) InfiniteBench tasks, GPT-4-turbo-2024-04-09 scores 33.16, Qwen2-72b-instruct 39.77, and Llama-3.1-70b-instruct 39.81. On 32k tasks, GPT-4-turbo-2024-04-09 leads at 51.93, ahead of Qwen2-72b-instruct (49.94) and Llama-3.1-70b-instruct (49.92). On 4k ChatRAG tasks, GPT-4-turbo-2024-04-09 scores 54.72, Qwen2-72b-instruct 54.06, and Llama-3.1-70b-instruct 52.12. The ranking shifts across context lengths: GPT-4-turbo is strongest at 32k but weakest at ultra-long among the top three.
The authors note that Qwen2-72b-instruct and Llama-3.1-70b-instruct used extensive 32k pretraining and large SFT data, while the authors used a smaller continued pretraining corpus, which may explain the 32k gap.