Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
SysBench: Can LLMs Follow System Message?
2025-01-22
· ICLR 2025 Poster ·
anchor
Findings
IC-188
LLMs show constraint-type-specific performance on system message following, with weaker models exhibiting large variance across constraint categories
IC-189
Most LLMs show degraded instruction satisfaction when user instructions conflict with system messages, indicating difficulty in prioritizing system message constraints
IC-190
LLMs show progressive degradation in system message constraint following across multi-turn conversations, with dependent conversations degrading faster than parallel ones
IC-191
Attention allocated to system messages correlates with following ability, and models do not strictly distinguish system from user messages based on marker tokens