25 Jan 2024 | Abeba Birhane, Ryan Steed, Victor Ojewale, Briana Vecchione, Inioluwa Deborah Raji
The article discusses the challenges and complexities of AI auditing, a practice aimed at ensuring accountability in AI systems. It examines the various stakeholders involved in AI auditing, including regulators, law firms, civil society, journalism, academia, consulting agencies, and government bodies. The study categorizes AI audit practices into four main types: product/model/algorithm audits, data audits, ecosystem audits, and meta-commentary. It finds that only a subset of AI audit studies effectively contribute to accountability outcomes. The study highlights the importance of audit design, methodology, and institutional context in determining the effectiveness of AI audits. It also notes that while academic audits often focus on theoretical or abstract issues, non-academic audits, such as those conducted by law firms, consulting agencies, and journalism, tend to have more tangible impacts. The study concludes that AI audits must be designed with clear objectives and methods to ensure they contribute to meaningful accountability. It also emphasizes the need for audits to be independent, transparent, and inclusive of diverse perspectives to ensure they address the broader societal implications of AI. The study provides a comprehensive overview of the current state of AI auditing, highlighting the challenges and opportunities in this rapidly evolving field.The article discusses the challenges and complexities of AI auditing, a practice aimed at ensuring accountability in AI systems. It examines the various stakeholders involved in AI auditing, including regulators, law firms, civil society, journalism, academia, consulting agencies, and government bodies. The study categorizes AI audit practices into four main types: product/model/algorithm audits, data audits, ecosystem audits, and meta-commentary. It finds that only a subset of AI audit studies effectively contribute to accountability outcomes. The study highlights the importance of audit design, methodology, and institutional context in determining the effectiveness of AI audits. It also notes that while academic audits often focus on theoretical or abstract issues, non-academic audits, such as those conducted by law firms, consulting agencies, and journalism, tend to have more tangible impacts. The study concludes that AI audits must be designed with clear objectives and methods to ensure they contribute to meaningful accountability. It also emphasizes the need for audits to be independent, transparent, and inclusive of diverse perspectives to ensure they address the broader societal implications of AI. The study provides a comprehensive overview of the current state of AI auditing, highlighting the challenges and opportunities in this rapidly evolving field.