SourceContrastive Learning is Spectral Clustering on Similarity Graph
The paper proves (Theorem 4.2) that the contrastive loss used to train CLIP on image-text pairs is exactly equivalent to running generalized spectral clustering on the bipartite graph whose edges connect each image to its paired text. The proof extends the SimCLR-to-spectral-clustering equivalence (Theorem 3.1) to the multi-modal setting by treating the pair graph as a directed bipartite graph with normalized out-degrees. Because the pair graph Laplacian is not necessarily symmetric, the result is called 'generalized' spectral clustering. The authors note that in practice CLIP uses a batch size of 32,768 and uniform edge sampling, which differs slightly from the theorem's uniform object sampling, but the difference is negligible when the dataset has high quality and low variance in object out-degrees.