IC-936GPT-2's learned positional encodings cause context vectors to lose linear independence after one layer, whereas BERT's sinusoidal encodings preserve it

Sadegh Mahdavi, Renjie Liao, Christos Thrampoulidis

SourceMemorization Capacity of Multi-Head Attention in Transformers

The paper tests whether context vectors remain linearly independent (assumption 2) at the embedding layer (l0) and after one attention layer (l1) in pre-trained BERT and GPT-2 on Wikipedia text. In BERT, sinusoidal positional encodings preserve full rank at both depths. In GPT-2, the learned positional encodings in the pre-trained model have lower rank, causing assumption 2 to fail at l1 while it holds at l0. General position fails in all four model/depth combinations. The Kruskal rank of query vectors in trained BERT is 291 (below the context size 512), and in random BERT is 349.

Evidence

observational

Key metric

n/d ratio 512/1024 for BERT; Kruskal rank 291 (trained BERT) and 349 (random BERT); assumption 2: GPT-2 trained l0 ✓, l1 ×; BERT trained l0 ✓, l1 ✓; general position × in all cases

Caveat

The paper notes that 'linear independence assumptions are not well-suited for discrete data'; the Wikipedia text is tokenized, making the continuous linear independence test an approximation for a discrete setting.

Model

GPT-2, BERT

Concepts

Depth-dependent structure

Datasets

Wikipedia [eval]

Related findings

IC-935

Extraction

automatic-extraction