IC-645CLIP, PickScore, and HPSv2 text embeddings share a common direction (cone effect) that captures text-irrelevant preferences, and the orthogonal component c⊥p better measures T2I alignment; CLIP's untrained common direction makes it ineffective for reward fine-tuning

Luping Liu, Chao Du, Tianyu Pang, Zehan Wang, Chongxuan Li, Dong Xu

SourceImproving Long-Text Alignment for Text-to-Image Diffusion Models

The paper computes the mean text embedding direction v across a large prompt set and projects each text embedding onto it. All three CLIP-based models show a strong positive projection: η > 0.4 for CLIP and η > 0.6 for PickScore and HPSv2. Decomposing the text embedding into the component along v (text-irrelevant) and the orthogonal component c⊥p (text-relevant) reveals that c⊥p substantially improves T2I alignment retrieval: PickScore r@1 jumps from 42.34 to 67.94 (single) and HPSv2 from 54.00 to 67.60. The paper further shows that CLIP's common direction v is not trained on human preferences (unlike PickScore and HPSv2), so using CLIP as a reward signal for diffusion fine-tuning produces an undefined optimization direction and poor generation quality.

Evidence

correlational

Key metric

η > 0.4 for CLIP, η > 0.6 for PickScore and HPSv2; r@1 retrieval (single/average): CLIP-H cp 86.10/80.40 vs c⊥p 85.80/85.14, PickScore cp 42.34/16.72 vs c⊥p 67.94/64.28, HPSv2 cp 54.00/31.84 vs c⊥p 67.60/64.00

Caveat

The common direction v is computed from a specific prompt dataset; the paper notes the cone effect results from both model initialization and contrastive training, so the exact direction may vary with training details.

Model

CLIP / CLIP-ViT (LC), PickScore, HPSv2

Concepts

Linear representation

Methods

DRTune [supporting], LoRA [supporting]

Related work

Cone effect (Gao et al. 2019) [context]

Extraction

automatic-extraction