When the number of recent tokens in the StreamingLLM rolling cache is increased from 252 to 4092 (with 4 attention sink tokens held fixed), language modeling perplexity on 400k tokens of the PG19 test set does not decrease monotonically. For MPT-7B, perplexity worsens from 14.12 to 14.99 as the cache grows. For LLaMA-2-7B, perplexity improves to 9.08 at 2044 recent tokens but degrades to 9.59 at 4092. For Falcon-7B, perplexity improves to 12.34 at 1020 tokens but degrades to 12.84 at 2044. The authors attribute this to the models not fully utilizing the available context within the cache.
Evaluated only in the StreamingLLM setup with repositioned tokens; the non-monotonicity may partly reflect the positional reassignment mechanism rather than an inherent model limitation. The authors note this as a 'potential limitation' and call for future work.