IC-016Truncating MLP weights in Pythia-1b increases the probability of the correct answer in a factual recall task

Lei Chen, Joan Bruna, Alberto Bietti

SourceDistributional Associations vs In-Context Reasoning: A Study of Feed-forward and Attention Layers

The paper evaluates the effect of LASER on the performance of Pythia-1b on a factual recall task. They measure the average probability of predicting the target fact ('spain') and the generic token 'the' for the prompt 'madrid is located in'. Applying LASER to the MLP weights of Pythia-1b boosts the probability ratio of 'spain' over 'the' from 0.16x to 11.3x at 14k training steps.

Evidence

correlational

Key metric

Average probability ratio of 'spain' over 'the' improves from 0.16x to 11.3x at 14k steps.

Caveat

The paper notes that better prompting could avoid the need for LASER in this case.

Model

Pythia

Methods

Layer-Selective Rank Reduction / LASER [primary]

Extraction

automatic-extraction