IC-161Linear probes on Pythia-70m and Gemma-2-2b trained on the ambiguous Bias in Bios set rely on gender as a spurious feature, with gender accuracy far exceeding profession accuracy
Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, Aaron Mueller
The authors train linear classification heads on Pythia-70m and Gemma-2-2b to classify profession from biographies, using only the ambiguous training set where gender is perfectly predictive of the label. On the balanced test set, the original probes achieve much higher gender accuracy than profession accuracy (87.4% vs. 61.9% for Pythia; 81.9% vs. 67.7% for Gemma), indicating the models' representations encode gender in a way that dominates profession classification. After applying SHIFT (ablating human-identified gender-related SAE features) and retraining the probe, gender accuracy drops to near chance (52.0%, 52.4%) while profession accuracy rises to 93.1% and 95.0%.
Evidence
correlational
Key metric
Original: Pythia profession 61.9%, gender 87.4%, worst group 24.4%; Gemma profession 67.7%, gender 81.9%, worst group 18.2%. After SHIFT+retrain: Pythia profession 93.1%, gender 52.0%, worst group 89.0%; Gemma profession 95.0%, gender 52.4%, worst group 92.9%.
Caveat
The interpretability step (identifying which features to ablate) uses additional unlabeled data and human judgement; feature labels may vary across annotators. For Gemma, the feature ablations initially damage model performance, which is restored by retraining the linear head.