The paper explores the concept of AI red-teaming in the context of generative AI (GenAI) and its role in addressing safety, security, and trustworthiness concerns. While AI red-teaming is often promoted as a key strategy for identifying and mitigating risks, the paper highlights significant ambiguity in its definition, scope, and application. The authors analyze recent AI red-teaming activities and conduct a comprehensive survey of relevant research to characterize the practices and criteria for AI red-teaming. They find that prior methods and practices of AI red-teaming vary along several axes, including the purpose of the activity, the artifact under evaluation, the setting in which the activity is conducted, and the resulting decisions it informs. The paper argues that while red-teaming may be a valuable idea for characterizing GenAI harm mitigations, its use as a panacea for all regulatory concerns about model safety may verge on security theater. The authors propose a question bank to guide and scaffold future AI red-teaming practices and suggest that red-teaming should be considered as one evaluation paradigm among others to assess and improve the safety and trustworthiness of GenAI. The paper also highlights the lack of consensus on the scope, structure, and assessment criteria for AI red-teaming, as well as the need for more concrete definitions and guidelines. The authors emphasize the importance of diverse perspectives and stakeholder involvement in evaluating GenAI systems and the need for clear definitions and standards for red-teaming activities. The paper concludes that red-teaming should be considered as part of a broader evaluation toolbox, complementing other approaches to ensure the safety and trustworthiness of GenAI.The paper explores the concept of AI red-teaming in the context of generative AI (GenAI) and its role in addressing safety, security, and trustworthiness concerns. While AI red-teaming is often promoted as a key strategy for identifying and mitigating risks, the paper highlights significant ambiguity in its definition, scope, and application. The authors analyze recent AI red-teaming activities and conduct a comprehensive survey of relevant research to characterize the practices and criteria for AI red-teaming. They find that prior methods and practices of AI red-teaming vary along several axes, including the purpose of the activity, the artifact under evaluation, the setting in which the activity is conducted, and the resulting decisions it informs. The paper argues that while red-teaming may be a valuable idea for characterizing GenAI harm mitigations, its use as a panacea for all regulatory concerns about model safety may verge on security theater. The authors propose a question bank to guide and scaffold future AI red-teaming practices and suggest that red-teaming should be considered as one evaluation paradigm among others to assess and improve the safety and trustworthiness of GenAI. The paper also highlights the lack of consensus on the scope, structure, and assessment criteria for AI red-teaming, as well as the need for more concrete definitions and guidelines. The authors emphasize the importance of diverse perspectives and stakeholder involvement in evaluating GenAI systems and the need for clear definitions and standards for red-teaming activities. The paper concludes that red-teaming should be considered as part of a broader evaluation toolbox, complementing other approaches to ensure the safety and trustworthiness of GenAI.