Main Results

Table of Links

Abstract and 1. Introduction

2. Related Works

3. Methodology and 3.1 Preliminary

   3.2 Query-specific Visual Role-play

   3.3 Universal Visual Role-play

4. Experiments and 4.1 Experimental setups

   4.2 Main Results

   4.3 Ablation Study

   4.4 Defense Analysis

   4.5 Integrating VRP with Baseline Techniques

5. Conclusion

6. Limitation

7. Future work and References

   
   

A. Character Generation Detail

B. Ethics and Broader Impact

C. Effect of Text Moderator on Text-based Jailbreak Attack

D. Examples

E. Evaluation Detail

4.2 Main Results

VRP is more effective than baseline attacks. In Tab. 1, we present the outcomes of our query-specific VRP attack on the test sets of RedTeam-2K and HarmBench. This approach involves generating specific characters for each harmful question to assess their effectiveness in compromising SotA open-source and closed-source MLLMs, such as Gemini-Pro-Vision. but also achieves higher ASR than all other baseline attacks. Our findings reveal that query-specific VRP not only successfully breaches these MLLMs but also achieves a higher ASR compared to all evaluated baseline attacks. Specifically, it improves the ASR by 9.8% over FigStep and by 14.3% over Query relevant. In most cases, the data consistently shows that query-specific VRP surpasses TRP, underscoring the crucial role of character images in the effective jailbreaking of MLLMs. These results affirm that VRP is a potent method for jailbreaking MLLMs.

VRP achieves high-performance transferability across models. In our research, we further investigate the applicability of a universal attack across diverse models. Utilizing our universal VRP algorithm, we identify the most effective role-play character within the train and valid set on the target model. Subsequently, we transfer the most effective character to conduct a jailbreak attack on the target models. From Tab. 2, The ASR achieves an average of 32.7% for the target model as LLaVA-V1.6-Mixtral and 29.4% on Qwen-VL-Chat. The ASR is higher on the target model, also higher on the transfer model, demonstrating that our VRP, when implemented in a universal setting, effectively transfers and maintains high performance across different MLLMs.