SourceSysBench: Can LLMs Follow System Message?
For three open-source models (GLM-4-9b, Llama3.1-8b, Qwen2-72b), the paper finds a strong correlation between the proportion of attention scores attributed to system messages and the model's ability to follow them. The attention proportion declines over turns, matching the Rn degradation pattern. Additionally, when system message text is repurposed as a user message, the attention score changes are described as 'very weak,' indicating the model does not use the marker token to create a strict distinction between system and user messages during inference. The authors conclude that system message following ability is more influenced by training data construction than by the inference-time marker.