Winogender

anchor

Findings

• IC-1549All 28 evaluated LMs exhibit gender bias on non-stereotypical sentence pairs, with fairness scores between 9% and 41% [builds-on]
• IC-1549All 28 evaluated LMs exhibit gender bias on non-stereotypical sentence pairs, with fairness scores between 9% and 41% [eval]
• IC-1550All evaluated LMs systematically prefer male pronoun completions in the non-stereotypical portions of Winobias and Winogender, with margins exceeding 40% [eval]
• IC-1551No consistent relationship between model size and gender fairness scores is observed across six LM families [eval]
• IC-1552Deduplication of pretraining data does not consistently improve gender fairness in Pythia models [eval]