Prompts, Pearls, Imperfections: Comparing ChatGPT and a Human Researcher in Qualitative Data Analysis

Prompts, Pearls, Imperfections: Comparing ChatGPT and a Human Researcher in Qualitative Data Analysis

2024 | Jonas Wachinger, Kate Bärnighausen, Louis N. Schäfer, Kerry Scott, and Shannon A. McMahon
This article explores the potential and limitations of ChatGPT in qualitative data analysis, comparing its performance with that of an experienced human researcher. The study focuses on analyzing an interview transcript about AI in medical practice using various prompts and analytic approaches. Key findings include: 1. **Descriptive Insights**: ChatGPT provided descriptive insights with considerable face validity, aligning with themes identified by the human researcher. 2. **Interpretative Contributions**: ChatGPT demonstrated the ability to propose interpretative insights, such as linking identified themes to broader theoretical frameworks. 3. **Codebook and Quotes**: ChatGPT generated a codebook and key quotes from the transcript, which had face validity but required careful review. 4. **Theoretical Engagement**: ChatGPT successfully embedded findings into broader theoretical discourses, even when the models seemed unfitting. 5. **Challenges and Limitations**: Despite its strengths, ChatGPT faced challenges in interpreting non-verbal cues and adapting to different analytic approaches without human guidance. 6. **Ethical Considerations**: The use of ChatGPT in academic research raises ethical concerns, particularly regarding data protection and confidentiality. 7. **Ongoing Developments**: The rapid evolution of generative AI applications suggests that future research should explore the potential of different tools and prompts. The authors conclude that while ChatGPT can provide valuable support in qualitative analysis, it must be used with caution and human oversight to ensure rigor and avoid biases. They call for further research to understand the conditions under which ChatGPT can effectively assist in qualitative research.This article explores the potential and limitations of ChatGPT in qualitative data analysis, comparing its performance with that of an experienced human researcher. The study focuses on analyzing an interview transcript about AI in medical practice using various prompts and analytic approaches. Key findings include: 1. **Descriptive Insights**: ChatGPT provided descriptive insights with considerable face validity, aligning with themes identified by the human researcher. 2. **Interpretative Contributions**: ChatGPT demonstrated the ability to propose interpretative insights, such as linking identified themes to broader theoretical frameworks. 3. **Codebook and Quotes**: ChatGPT generated a codebook and key quotes from the transcript, which had face validity but required careful review. 4. **Theoretical Engagement**: ChatGPT successfully embedded findings into broader theoretical discourses, even when the models seemed unfitting. 5. **Challenges and Limitations**: Despite its strengths, ChatGPT faced challenges in interpreting non-verbal cues and adapting to different analytic approaches without human guidance. 6. **Ethical Considerations**: The use of ChatGPT in academic research raises ethical concerns, particularly regarding data protection and confidentiality. 7. **Ongoing Developments**: The rapid evolution of generative AI applications suggests that future research should explore the potential of different tools and prompts. The authors conclude that while ChatGPT can provide valuable support in qualitative analysis, it must be used with caution and human oversight to ensure rigor and avoid biases. They call for further research to understand the conditions under which ChatGPT can effectively assist in qualitative research.
Reach us at info@study.space
[slides] Prompts%2C Pearls%2C Imperfections%3A Comparing ChatGPT and a Human Researcher in Qualitative Data Analysis. | StudySpace