August 12, 2024 | Steve Rathjea,1,2, Dan-Mircea Mirea1,2, Ilia Sucholutsky1, Raja Marjieh1, Claire E. Robertson1, and Jay J. Van Bavel1,2,6,7
The study explores the effectiveness of GPT, a large language model (LLM), in automated psychological text analysis across multiple languages. Using 15 datasets containing 47,925 manually annotated tweets and news headlines, the researchers tested GPT ( versions 3.5 Turbo, 4, and 4 Turbo) for detecting psychological constructs such as sentiment, discrete emotions, offensiveness, and moral foundations. The results show that GPT performed significantly better than English-language dictionary analysis (r = 0.20 to 0.30) and was nearly as effective as, or sometimes better than, several top-performing fine-tuned machine learning models. GPT's performance improved with successive model versions, particularly for lesser-spoken languages, and it required no training data or extensive coding experience. The study also highlights GPT's potential to democratize automated text analysis by making advanced natural language processing capabilities more accessible and facilitating cross-linguistic research with understudied languages. Overall, GPT is presented as a superior tool for automated text analysis due to its high accuracy, ease of use, and broad applicability across languages.The study explores the effectiveness of GPT, a large language model (LLM), in automated psychological text analysis across multiple languages. Using 15 datasets containing 47,925 manually annotated tweets and news headlines, the researchers tested GPT ( versions 3.5 Turbo, 4, and 4 Turbo) for detecting psychological constructs such as sentiment, discrete emotions, offensiveness, and moral foundations. The results show that GPT performed significantly better than English-language dictionary analysis (r = 0.20 to 0.30) and was nearly as effective as, or sometimes better than, several top-performing fine-tuned machine learning models. GPT's performance improved with successive model versions, particularly for lesser-spoken languages, and it required no training data or extensive coding experience. The study also highlights GPT's potential to democratize automated text analysis by making advanced natural language processing capabilities more accessible and facilitating cross-linguistic research with understudied languages. Overall, GPT is presented as a superior tool for automated text analysis due to its high accuracy, ease of use, and broad applicability across languages.