SourceTraining-Free Activation Sparsity in Large Language Models
When varying the proportion of prefill tokens sparsified along the sequence dimension, the paper finds that sparsifying the second half of prefill is nearly identical to sparsifying 99% of prefill (all tokens except the initial ones). However, including the initial tokens in sparsification causes substantially more perplexity degradation. The authors attribute this to the attention sink phenomenon, where initial tokens receive outsized attention due to the softmax operation, making their keys and values disproportionately important to the model's output.