IC-188LLMs show constraint-type-specific performance on system message following, with weaker models exhibiting large variance across constraint categories
Yanzhao Qin, Tao Zhang, Yanjun Shen, Wenjing Luo, sunhaoze, Yan Zhang, Yujing Qiao, weipeng chen, Zenan Zhou, Wentao Zhang, Bin CUI
The paper evaluates 16 LLMs on 500 system messages containing six constraint types (action, content, background, role, format, style). GPT-4o leads across all types with an overall CSR of 87.1%. Stronger models show relatively uniform performance across constraint types, while weaker models show significant variance. For example, Qwen2-7b achieves 81.0% on role constraints but drops to 43.3% on style constraints. ERNIE-4, despite performing well on other instruction-following benchmarks, scores only 50.7% overall on system message following, comparable to Qwen2-7b.
Evidence
correlational
Key metric
GPT-4o CSR 87.1% (action 86.8%, content 86.9%, background 87.2%, role 93.5%, format 87.4%, style 86.5%); Qwen2-7b role 81.0% vs. style 43.3%; ERNIE-4 total 50.7%
Caveat
Evaluation uses GPT-4o as a model-based verifier; the paper reports 97.3% human-model consistency at constraint granularity but this is a proxy for ground truth.