Layer-Selective Rank Reduction / LASER

anchor

Replace selected weight matrices with low-rank approximations and measure where accuracy rises rather than falls.

Findings

• IC-015Truncating MLP weights in Pythia-1b increases the probability of the correct answer in an Indirect Object Identification task [primary]
• IC-016Truncating MLP weights in Pythia-1b increases the probability of the correct answer in a factual recall task [primary]
• IC-017Truncating MLP weights improves few-shot Chain-of-Thought reasoning accuracy on GSM8K for Phi-3 and Llama-3.1-8B [primary]
• IC-072LLM agents of varying scales exhibit a failure mode on web automation tasks when processing raw, complex web page observations, with the penalty being more severe for smaller models [compared-to]