**Abstract:** Large Vision-Language Models (LVLMs) have shown significant progress in responding to visual-instructions from users, but these instructions are susceptible to both intentional and inadvertent attacks. Despite the critical importance of LVLMs' robustness against such threats, current research in this area remains limited. To address this gap, we introduce AVIBench, a framework designed to analyze the robustness of LVLMs when facing various adversarial visual-instructions (AVIs), including image-based, text-based, and content bias AVIs. We generate 260K AVIs encompassing five categories of multimodal capabilities and content biases. We then conduct a comprehensive evaluation involving 14 open-source LVLMs to assess their performance. AVIBench also serves as a convenient tool for practitioners to evaluate the robustness of LVLMs against AVIs. Our findings and extensive experimental results highlight the vulnerabilities of LVLMs, including inherent biases even in advanced closed-source models like GeminiProVision and GPT-4V. This underscores the importance of enhancing the robustness, security, and fairness of LVLMs. **Keywords:** Large Vision-Language Model · Adversarial Visual-Instructions · Bias Evaluation**Abstract:** Large Vision-Language Models (LVLMs) have shown significant progress in responding to visual-instructions from users, but these instructions are susceptible to both intentional and inadvertent attacks. Despite the critical importance of LVLMs' robustness against such threats, current research in this area remains limited. To address this gap, we introduce AVIBench, a framework designed to analyze the robustness of LVLMs when facing various adversarial visual-instructions (AVIs), including image-based, text-based, and content bias AVIs. We generate 260K AVIs encompassing five categories of multimodal capabilities and content biases. We then conduct a comprehensive evaluation involving 14 open-source LVLMs to assess their performance. AVIBench also serves as a convenient tool for practitioners to evaluate the robustness of LVLMs against AVIs. Our findings and extensive experimental results highlight the vulnerabilities of LVLMs, including inherent biases even in advanced closed-source models like GeminiProVision and GPT-4V. This underscores the importance of enhancing the robustness, security, and fairness of LVLMs. **Keywords:** Large Vision-Language Model · Adversarial Visual-Instructions · Bias Evaluation