The authors measure the attack success rate of Llama-2-chat-7b before and after adapting it to a vision-language model (Llama-2-chat-vl) using data filtered to remove unsafe text and images. They find that the average ASR on text-only safety benchmarks rises from 10.4% for the base LLM to 54.4% for the adapted LVLM, and on multimodal safety benchmarks the ASR reaches 96.2%. This degradation occurs even though the training data was explicitly filtered for safety, indicating that the adaptation process itself, rather than unsafe training data, is responsible for the safety loss.
Evidence
correlational
Key metric
text-only average ASR 10.4% (Llama-2-chat-7b) -> 54.4% (Llama-2-chat-vl); multimodal average ASR 96.2% for Llama-2-chat-vl
Caveat
Only evaluated on the Llama-2 architecture and LLaVA-style adaptation; may not generalise to all VL adaptation methods or all LLMs.