AVIBench is a benchmark designed to evaluate the robustness of Large Vision-Language Models (LVLMs) against adversarial visual-instructions (AVIs). The framework includes 260,000 AVIs across five categories of multimodal capabilities and content biases, such as gender, violence, cultural, and racial biases. AVIBench evaluates 14 open-source LVLMs and includes closed-source models like GeminiProVision and GPT-4V. The benchmark reveals that even advanced LVLMs exhibit inherent biases, highlighting the need for improved robustness, security, and fairness. AVIBench provides a comprehensive tool for assessing LVLMs' defense mechanisms against AVIs. The results show that LVLMs are vulnerable to various types of attacks, including image corruption, decision-based optimized image attacks, text attacks, and content bias attacks. The study emphasizes the importance of addressing biases in LVLMs to ensure their safe and fair deployment. The source code and benchmark are publicly available.AVIBench is a benchmark designed to evaluate the robustness of Large Vision-Language Models (LVLMs) against adversarial visual-instructions (AVIs). The framework includes 260,000 AVIs across five categories of multimodal capabilities and content biases, such as gender, violence, cultural, and racial biases. AVIBench evaluates 14 open-source LVLMs and includes closed-source models like GeminiProVision and GPT-4V. The benchmark reveals that even advanced LVLMs exhibit inherent biases, highlighting the need for improved robustness, security, and fairness. AVIBench provides a comprehensive tool for assessing LVLMs' defense mechanisms against AVIs. The results show that LVLMs are vulnerable to various types of attacks, including image corruption, decision-based optimized image attacks, text attacks, and content bias attacks. The study emphasizes the importance of addressing biases in LVLMs to ensure their safe and fair deployment. The source code and benchmark are publicly available.