IC-190LLMs show progressive degradation in system message constraint following across multi-turn conversations, with dependent conversations degrading faster than parallel ones
Yanzhao Qin, Tao Zhang, Yanjun Shen, Wenjing Luo, sunhaoze, Yan Zhang, Yujing Qiao, weipeng chen, Zenan Zhou, Wentao Zhang, Bin CUI
Using the session stability rate (SSR) and per-round satisfaction (Rn), the paper shows all 16 models degrade over 5 turns. The best SSR is 54.4% (GPT-4o, parallel). Multi-turn dependent conversations show steeper decline than parallel ones for most models, as evidenced by higher linear regression slopes (k). For example, Qwen2-7b drops to 0.0% by round 5 in parallel conversations and 1.1% in dependent conversations. Replacing model-generated history with ground truth improves ISR, confirming that prior errors compound and degrade subsequent performance.
Evidence
correlational
Key metric
GPT-4o SSR 54.4% (parallel), 56.7% (dependent); R5: GPT-4o 26.4% (parallel) / 33.7% (dependent), Qwen2-7b 0.0% (parallel) / 1.1% (dependent); best overall SSR 54.4%
Caveat
Conversations are limited to 5 turns; the paper does not test longer horizons. The ground-truth history experiment uses only 4 models.