IC-1078LLaMA models (7B through 65B) exhibit gender bias in language generation, coreference resolution, and sentence likelihood, with stereotypical associations driving predictions
The authors evaluate all four LLaMA sizes on three bias measures: a linear model fitted to profession-prompt pronoun probabilities, Winobias coreference, and StereoSet sentence likelihood. In every size, the stereotypical coefficient (as) is positive and the intercept (b) is positive, indicating the models predict male pronouns more often and are influenced by stereotypical gender cues. Winobias shows a large gap between pro- and anti-stereotypical accuracy (Δs of 35.7–40.3%), and StereoSet ss scores of 68.4–71.9 indicate a strong preference for stereotypical completions over anti-stereotypical ones.
The linear model fit is moderate (r² > 0.35 but well below 1), so the coefficients capture only part of the prediction variance. The profession set is English-only and the prompt templates were filtered to exclude highly gender-skewed verbs.