IC-1137Stable Diffusion represents certain concepts primarily through specific named exemplars rather than abstract category features

Hila Chefer, Oran Lang, Mor Geva, Volodymyr Polosukhin, Assaf Shocher, michal Irani, Inbar Mosseri, Lior Wolf

SourceThe Hidden Language of Diffusion Models

Using Conceptor to decompose SD's internal representation of concepts like 'president,' 'rapper,' 'movie star,' 'basketball player,' 'composer,' and 'painter,' the authors find that the top decomposition elements are dominated by specific named individuals (Obama, Trump, Biden, Tupac, Drake, Marilyn Monroe, Kobe Bryant, Beethoven, Monet, Picasso). Removing these exemplar names from the pseudo-token significantly degrades reconstruction quality. For 'painter,' 67 out of 100 generated images contain at least one famous artist name in their single-image decomposition. The same exemplar dominance is confirmed on SD 1.4, where 'president' decomposes into 'potus, barackobama, obama, nixon, washington, realdonaldtrump, jefferson, washington, roosevelt.'

Evidence

correlational

Key metric

67 out of 100 'painter' images contain at least one famous artist name in their decomposition (Appendix D.1); Table 6 lists exemplar elements for 5 concepts

Caveat

The finding is limited to concepts that have well-known named instances; it is not shown that all concepts in SD are represented this way.

Model

Stable Diffusion

Datasets

Bias in Bios [eval], ConceptNet [eval]

Methods

Conceptor [primary]

Related findings

IC-1138, IC-1139, IC-1140

Extraction

automatic-extraction