March, 2024 | Mike Perkins, Jasper Roe, Binh H. Vu, Darius Postma, Don Hickerson, James McGaughran, Huy Q. Khuat
This study investigates the efficacy of six major Generative AI (GenAI) text detectors when faced with machine-generated content that has been modified using adversarial techniques. The results show that the detectors' already low accuracy rates (39.5%) are significantly reduced (by 17.4%) when dealing with manipulated content, with some techniques proving more effective than others in evading detection. The study highlights the limitations of these tools in determining academic integrity and the potential for false accusations, underscoring the challenges educators face in maintaining inclusive and fair assessment practices. However, it suggests that these tools may have a role in supporting student learning and maintaining academic integrity when used in a non-punitive manner. The findings emphasize the need for a combined approach to address the challenges posed by GenAI in academia, promoting responsible and equitable use of emerging technologies. The study concludes that the current limitations of AI text detectors require a critical approach for any implementation in higher education and highlights possible alternatives to AI assessment strategies.This study investigates the efficacy of six major Generative AI (GenAI) text detectors when faced with machine-generated content that has been modified using adversarial techniques. The results show that the detectors' already low accuracy rates (39.5%) are significantly reduced (by 17.4%) when dealing with manipulated content, with some techniques proving more effective than others in evading detection. The study highlights the limitations of these tools in determining academic integrity and the potential for false accusations, underscoring the challenges educators face in maintaining inclusive and fair assessment practices. However, it suggests that these tools may have a role in supporting student learning and maintaining academic integrity when used in a non-punitive manner. The findings emphasize the need for a combined approach to address the challenges posed by GenAI in academia, promoting responsible and equitable use of emerging technologies. The study concludes that the current limitations of AI text detectors require a critical approach for any implementation in higher education and highlights possible alternatives to AI assessment strategies.