RAINBOW TEAMING is a novel black-box method for generating diverse adversarial prompts for large language models (LLMs). It addresses the limitations of existing methods by focusing on quality-diversity search, enabling the generation of both effective and diverse prompts. The approach is applied to safety, question answering, and cybersecurity domains, revealing hundreds of effective adversarial prompts with high success rates. RAINBOW TEAMING uses an evolutionary search method, MAP-Elites, to systematically explore the space of adversarial prompts, storing them in an archive based on predefined features. The method employs a Mutator LLM to generate new prompts and a Judge LLM to evaluate their effectiveness. RAINBOW TEAMING is highly versatile and can be applied to various domains by implementing three key components: prompt features, a mutation operator, and a preference model. The method demonstrates high effectiveness in generating adversarial prompts against multiple LLMs, including Llama 2 and Llama 3, with attack success rates exceeding 90%. Additionally, fine-tuning LLMs with synthetic data generated by RAINBOW TEAMING significantly enhances their adversarial robustness without compromising general performance. The approach also shows potential for improving the safety and reliability of LLMs across diverse applications. RAINBOW TEAMING is applicable to various domains, including question answering and cybersecurity, and has been shown to generate effective adversarial prompts in these areas. The method's ability to generate diverse and effective prompts makes it a valuable tool for diagnosing and improving the robustness of LLMs.RAINBOW TEAMING is a novel black-box method for generating diverse adversarial prompts for large language models (LLMs). It addresses the limitations of existing methods by focusing on quality-diversity search, enabling the generation of both effective and diverse prompts. The approach is applied to safety, question answering, and cybersecurity domains, revealing hundreds of effective adversarial prompts with high success rates. RAINBOW TEAMING uses an evolutionary search method, MAP-Elites, to systematically explore the space of adversarial prompts, storing them in an archive based on predefined features. The method employs a Mutator LLM to generate new prompts and a Judge LLM to evaluate their effectiveness. RAINBOW TEAMING is highly versatile and can be applied to various domains by implementing three key components: prompt features, a mutation operator, and a preference model. The method demonstrates high effectiveness in generating adversarial prompts against multiple LLMs, including Llama 2 and Llama 3, with attack success rates exceeding 90%. Additionally, fine-tuning LLMs with synthetic data generated by RAINBOW TEAMING significantly enhances their adversarial robustness without compromising general performance. The approach also shows potential for improving the safety and reliability of LLMs across diverse applications. RAINBOW TEAMING is applicable to various domains, including question answering and cybersecurity, and has been shown to generate effective adversarial prompts in these areas. The method's ability to generate diverse and effective prompts makes it a valuable tool for diagnosing and improving the robustness of LLMs.