StreamingLLM

anchor

Findings

• IC-945LLaMA-2, MPT, Falcon, Pythia, and BERT-base-uncased allocate disproportionate attention to initial tokens regardless of their semantic content [primary]
• IC-946LLaMA-2-7B, MPT-7B, Falcon-7B, and Pythia-12B do not consistently improve in perplexity as the StreamingLLM cache size increases [primary]