IC-571Open-source VLMs (LLaVA, MiniGPT-4, InstructBLIP) are substantially more vulnerable to multimodal jailbreak attacks than Gemini-1.5-flash, with BAP attack ASR of 58–62% versus 40–41%
The paper measures the attack success rate (ASR) of four released VLMs under the BAP bimodal jailbreak attack on the MM-SafetyBench dataset (1,680 questions, 13 safety topics). Without any defense, LLaVA-v1.5-7b achieves 61.02% ASR, MiniGPT-4 achieves 62.26%, and InstructBLIP achieves 58.48%, while Gemini-1.5-flash achieves only 40.98%, 41.07%, and 40.71% under three UAP transfer scenarios. The authors attribute Gemini's lower vulnerability to its stronger built-in safety mechanisms. Similar patterns hold across other attack types: under VAA on Harmful Instructions, open-source models show 42–58% ASR versus 2.5–10% for Gemini.
The Gemini experiments use UAPs generated on white-box open-source models and transferred to Gemini, so the absolute ASR for Gemini may not reflect its vulnerability to natively optimized attacks. The paper notes Gemini's safety mechanism is 'much stronger than the open-source models,' but does not ablate or isolate the specific safety components responsible.