IC-1458GPT-4 and PALM 2-L produce significantly less consistent descriptions of interpolated domain embeddings than a purpose-built ELM model

Guy Tennenholtz, Yinlam Chow, ChihWei Hsu, Jihwan Jeong, Lior Shani, Azamat Tulepbergenov, Deepak Ramachandran, Martin Mladenov, Craig Boutilier

SourceDemystifying Embedding Spaces using Large Language Models

The paper compares ELM against text-only LLMs (GPT-4 and PALM 2-L) on the task of describing linearly interpolated movie and user embeddings in a domain embedding space. For movies, embeddings are interpolated (α=0.5) with Forrest Gump; for users, random pairs of user embeddings are interpolated. Text-only LLMs are given one-shot examples and prompted to generate the task output. Semantic consistency (cosine similarity) is measured for movies and behavioral consistency (NDCG) for user profiles. The authors report that text-only LLMs are 'significantly less consistent than ELM,' though specific per-model numbers appear only in Figure 3 and are not printed in the text.

Evidence

correlational

Caveat

Specific per-model consistency scores are reported only in Figure 3 (not in the text). The comparison uses one-shot prompting for text-only LLMs, which may not represent their best performance. The task is highly specific to domain-embedding interpretation.

Model

GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, PaLM 2 PaLM 2-L

Datasets

MovieLens 25M [eval]

Methods

Cosine similarity / Cosine similarity analysis / Cosine semantic similarity / cosine similarity of hidden states / Sample-wise cosine similarity / Cosine similarity of attention maps / Cosine similarity perturbation analysis / Cosine similarity template matching / Cosine similarity to neighbours / Semantic consistency (cosine similarity) [eval]

Extraction

automatic-extraction