Head masking

Zero out attention heads one at a time or in groups and measure the effect on behaviour, the pruning-style counterpart to activation patching.

Findings

• IC-004Retrieval heads are sparse, universal, and causally responsible for long-context retrieval in LLMs [primary]