IC-739GPT-3.5-turbo-0613 combined with CLIP produces more faithful concept-salience pseudo-labels than LLaMA-2-13B-Chat, InstructBLIP, or LLaVA-1.5B on most of five datasets
Aditya Chattopadhyay, Kwan Ho Ryan Chan, Rene Vidal
The paper compares four mechanisms for generating pseudo-labels indicating whether a semantic concept is salient for recognizing a given image class. GPT-3.5-turbo-0613 (with CLIP dot products) achieves 0.79–0.86 accuracy across ImageNet, Places365, CUB-200, CIFAR-10, and CIFAR-100. LLaMA-2-13B-Chat with CLIP is worse on most datasets (0.62 on CUB-200 vs 0.79 for GPT). InstructBLIP (FlanT5XL) is competitive with GPT+CLIP on most datasets but worse on CUB-200 (0.68 vs 0.79). LLaVA-1.5B is the weakest at 0.49–0.69. The authors note that GPT-4 was substituted for GPT-3.5 on CUB-200 due to domain-specificity, so that column may reflect GPT-4 performance.
Ground truth is self-annotated by the authors (10,000 evaluations over two weeks). The CUB-200 column for GPT+CLIP uses GPT-4 rather than GPT-3.5-turbo-0613, as stated in the paper. The authors note that using VLMs (InstructBLIP, LLaVA) to generate pseudo-labels for the full ImageNet training set is computationally infeasible (~37,344 GPU hours for InstructBLIP vs ~600 for GPT+CLIP).