IC-748Pruned LLMs at ≥50% sparsity remain robust in-context retrievers and summarizers, with Vicuna-7B matching up to ~40% sparsity and Vicuna-13B up to ~50% sparsity in open-book settings
AJAY KUMAR JAISWAL, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, Yinfei Yang
When external knowledge is provided in-context (open-book ICRA-QA on TriviaQA, and in-context summarization on CNN/DailyMail), pruned LLMs retain their ability to extract and synthesize information from the prompt. Vicuna-7B remains within the 5% matching tolerance up to ~40% unstructured sparsity and 8-bit quantization for retrieval, while Vicuna-13B holds up to ~50% sparsity and 4-bit quantization. For summarization, all pruning and quantization methods preserve coherence, consistency, fluency, and relevance, though longer stories show sharper degradation. This contrasts sharply with the closed-book knowledge loss observed in Finding 1.
Evidence
interventional
Key metric
Vicuna-7B matching up to ~40% sparsity and 8-bit quantization; Vicuna-13B matching up to ~50% sparsity and 4-bit quantization (open-book ICRA-QA); in-context retrieval coherence (long story): SparseGPT 5.883→5.067, Wanda 5.883→5.033 at 50% sparsity
Caveat
With increasing context length (large stories), performance drop is sharper; no matching compressed LLM identified for 2:4 structured sparsity; GPT-4 used as judge introduces potential evaluation bias