Prompts, Pearls, Imperfections: Comparing ChatGPT and a Human Researcher in Qualitative Data Analysis

Prompts, Pearls, Imperfections: Comparing ChatGPT and a Human Researcher in Qualitative Data Analysis

2024 | Jonas Wachinger, Kate Bärnighausen, Louis N. Schäfer, Kerry Scott, Shannon A. McMahon
This article explores the performance of ChatGPT in qualitative data analysis compared to that of an experienced human researcher. The study involved analyzing an interview transcript on AI in clinical practice, with ChatGPT prompted to identify themes and provide example quotes. The human researcher used Reflexive Thematic Analysis, while ChatGPT followed a variety of approaches, including Grounded Theory and the Five Step Framework. Both methods identified similar themes, though ChatGPT leaned toward descriptive themes and identified more nuanced dynamics such as 'trust and responsibility' and 'acceptance and resistance'. ChatGPT was able to propose a codebook and key quotes with considerable face validity but required careful review. When prompted to embed findings into broader theoretical discourses, ChatGPT could convincingly argue how identified themes linked to provided theories, even in cases of seemingly unfitting models. Despite challenges, ChatGPT performed better than expected, especially in identifying themes overlapping with those of an experienced researcher and embedding these themes into specific theoretical debates. The study discusses how ChatGPT could contribute to and challenge established best-practice approaches for rigorous and nuanced qualitative research and teaching. The article highlights the potential of ChatGPT in qualitative research, while also noting its limitations, such as its inability to move beyond semantic meaning and its tendency to overfit data to provided theories. The study emphasizes the need for critical reflection and ethical considerations when using AI in qualitative research. It also calls for further research on how ChatGPT's theme generation might be shaped by previous interactions with researchers on the same or similar data and topics. The article concludes with a call for ongoing discourse on the role of AI in qualitative research, emphasizing the importance of theory, rigor, and reflexivity in ensuring the quality and ethical integrity of qualitative research.This article explores the performance of ChatGPT in qualitative data analysis compared to that of an experienced human researcher. The study involved analyzing an interview transcript on AI in clinical practice, with ChatGPT prompted to identify themes and provide example quotes. The human researcher used Reflexive Thematic Analysis, while ChatGPT followed a variety of approaches, including Grounded Theory and the Five Step Framework. Both methods identified similar themes, though ChatGPT leaned toward descriptive themes and identified more nuanced dynamics such as 'trust and responsibility' and 'acceptance and resistance'. ChatGPT was able to propose a codebook and key quotes with considerable face validity but required careful review. When prompted to embed findings into broader theoretical discourses, ChatGPT could convincingly argue how identified themes linked to provided theories, even in cases of seemingly unfitting models. Despite challenges, ChatGPT performed better than expected, especially in identifying themes overlapping with those of an experienced researcher and embedding these themes into specific theoretical debates. The study discusses how ChatGPT could contribute to and challenge established best-practice approaches for rigorous and nuanced qualitative research and teaching. The article highlights the potential of ChatGPT in qualitative research, while also noting its limitations, such as its inability to move beyond semantic meaning and its tendency to overfit data to provided theories. The study emphasizes the need for critical reflection and ethical considerations when using AI in qualitative research. It also calls for further research on how ChatGPT's theme generation might be shaped by previous interactions with researchers on the same or similar data and topics. The article concludes with a call for ongoing discourse on the role of AI in qualitative research, emphasizing the importance of theory, rigor, and reflexivity in ensuring the quality and ethical integrity of qualitative research.
Reach us at info@study.space
[slides and audio] Prompts%2C Pearls%2C Imperfections%3A Comparing ChatGPT and a Human Researcher in Qualitative Data Analysis.