SourceReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models
The paper measures how many distinct neurons are activated across a sequence of generated tokens in OPT 6.7B. On average, about 50% of all FFN neurons are never used across the first 150 tokens of WikiText prompts. This aggregated sparsity is significantly higher than what would be expected from random per-token activation (the observed curve exceeds the random sparsity curve in Fig 7b), indicating a structured reuse of neurons. Exploiting this temporal pattern, the paper shows that reusing previously loaded weights for subsequent tokens causes only a slight perplexity increase, and when integrated into speculative decoding, yields a 1.27x speedup at gamma=16 versus 1.20x for random sparsity.