SourceINViTE: INterpret and Control Vision-Language Models with Text Explanations
In an appendix experiment, the paper applies an adversarial perturbation to a forest image that causes CLIP to misclassify it as beach. By interpreting each token at every layer with ViTE before and after the attack, they measure the percentage of tokens that shift from interpreting as 'forest' to 'beach.' The percentage of affected tokens increases sharply starting around layer 10, indicating the adversarial effect is concentrated in the later layers of the 13-layer transformer.