IC-191Attention allocated to system messages correlates with following ability, and models do not strictly distinguish system from user messages based on marker tokens
Yanzhao Qin, Tao Zhang, Yanjun Shen, Wenjing Luo, sunhaoze, Yan Zhang, Yujing Qiao, weipeng chen, Zenan Zhou, Wentao Zhang, Bin CUI
For three open-source models (GLM-4-9b, Llama3.1-8b, Qwen2-72b), the paper finds a strong correlation between the proportion of attention scores attributed to system messages and the model's ability to follow them. The attention proportion declines over turns, matching the Rn degradation pattern. Additionally, when system message text is repurposed as a user message, the attention score changes are described as 'very weak,' indicating the model does not use the marker token to create a strict distinction between system and user messages during inference. The authors conclude that system message following ability is more influenced by training data construction than by the inference-time marker.
Evidence
correlational
Caveat
Only three open-source models are analyzed for attention; the correlation is described qualitatively without a printed coefficient. The attention analysis is averaged across all heads and layers, potentially masking layer-specific effects.