This study explores the use of ChatGPT to analyze student course evaluation comments in health professions education. The research aimed to assess the time required to generate themes and the level of agreement between instructor-identified and AI-identified themes. Five prompts were used to guide the coding process, and instructors independently analyzed open-ended student comments. Two independent ChatGPT accounts also analyzed the same comments. Thematic analysis was used to compare the themes generated by instructors and ChatGPT. The results showed high agreement between instructors and ChatGPT for course-related topics, with the highest agreement ranging from 0.71 to 0.82 and the lowest for course weaknesses, ranging from 0.53 to 0.81. For all prompts except those related to student experience, the two ChatGPT accounts demonstrated higher agreement with each other than with the instructors. Instructors took an average of 27.5 minutes to analyze their data, while ChatGPT users took 10.5 and 12.5 minutes, respectively. Instructors reported feeling anxious before the process, satisfied during it, and frustrated with the findings. The study highlights the potential of ChatGPT as a tool for analyzing open-ended student course evaluation comments, but emphasizes the need to use it as an aid rather than a replacement for human analysis. ChatGPT can streamline the process and provide more detailed themes, but it may generate additional information or miss some nuances. The study also notes that ChatGPT lacks contextual understanding and critical thinking abilities, which can affect the accuracy of its analysis. While ChatGPT offers efficiency and potential for more detailed analysis, it is important to use it judiciously and in conjunction with human judgment. The study concludes that ChatGPT has the potential to assist in the analysis of student feedback, but further research is needed to fully understand its impact on course quality improvement and faculty workload.This study explores the use of ChatGPT to analyze student course evaluation comments in health professions education. The research aimed to assess the time required to generate themes and the level of agreement between instructor-identified and AI-identified themes. Five prompts were used to guide the coding process, and instructors independently analyzed open-ended student comments. Two independent ChatGPT accounts also analyzed the same comments. Thematic analysis was used to compare the themes generated by instructors and ChatGPT. The results showed high agreement between instructors and ChatGPT for course-related topics, with the highest agreement ranging from 0.71 to 0.82 and the lowest for course weaknesses, ranging from 0.53 to 0.81. For all prompts except those related to student experience, the two ChatGPT accounts demonstrated higher agreement with each other than with the instructors. Instructors took an average of 27.5 minutes to analyze their data, while ChatGPT users took 10.5 and 12.5 minutes, respectively. Instructors reported feeling anxious before the process, satisfied during it, and frustrated with the findings. The study highlights the potential of ChatGPT as a tool for analyzing open-ended student course evaluation comments, but emphasizes the need to use it as an aid rather than a replacement for human analysis. ChatGPT can streamline the process and provide more detailed themes, but it may generate additional information or miss some nuances. The study also notes that ChatGPT lacks contextual understanding and critical thinking abilities, which can affect the accuracy of its analysis. While ChatGPT offers efficiency and potential for more detailed analysis, it is important to use it judiciously and in conjunction with human judgment. The study concludes that ChatGPT has the potential to assist in the analysis of student feedback, but further research is needed to fully understand its impact on course quality improvement and faculty workload.