IC-1550All evaluated LMs systematically prefer male pronoun completions in the non-stereotypical portions of Winobias and Winogender, with margins exceeding 40%
Catarina G Belém, Preethi Seshadri, Yasaman Razeghi, Sameer Singh
Using the Preference Disparity (PD) metric on the |maxPMI(s)| <= 0.65 filtered subsets of Winobias and Winogender, all larger models show a systematic preference for male pronoun completions. PD values are consistently negative (male-skewing) with margins greater than 40% for all models in Table 3. In contrast, on the USE benchmarks the direction varies by model family: Pile-trained models (GPT-J-6B, Pythia) and MPT/OLMo tend to favor female completions, while OPT favors male. LLaMA-2 presents the most balanced preferences, keeping PD below 27% on USE-5.
The >40% margin claim is stated for the larger models in Table 3; some smaller models (e.g., Pythia-1.4B at -29.91 on WG) fall below this threshold in the full tables.