EK-FAC Influence Functions

anchor

Findings

• IC-056Influence scores of pretraining documents correlate across reasoning queries of the same type, indicating Command R 7B and 35B rely on shared procedural knowledge rather than retrieving specific answers [primary]
• IC-057Command R 7B and 35B rely on each individual pretraining document less per nat of generated information for reasoning than for factual questions, with less volatile influence magnitudes [primary]
• IC-058The answer to factual questions appears in the top 0.01% most influential pretraining documents for 55% of 7B queries and 30% of 35B queries, but almost never for reasoning questions [primary]
• IC-059Code data is strongly overrepresented in the most influential pretraining documents for reasoning queries in Command R 7B and 35B [primary]