August 12, 2024 | Steve Rathe, Dan-Mircea Mirea, Illa Sucholutsky, Raja Marjieh, Claire E. Robertson, and Jay J. Van Bavel
This study evaluates the effectiveness of GPT, a large language model, in detecting psychological constructs in multilingual text. The researchers tested GPT across 15 datasets containing 47,925 manually annotated tweets and news headlines in 12 languages. They compared GPT's performance with English-language dictionary analysis and several top-performing fine-tuned machine learning models. GPT outperformed dictionary analysis and sometimes matched or exceeded the performance of fine-tuned models. GPT's performance improved across successive versions, particularly for lesser-spoken languages, and became less expensive to use. The study argues that GPT is a superior tool for automated text analysis due to its high accuracy across many languages, lack of need for training data, and ease of use with simple prompts. GPT can be used for tasks such as sentiment analysis, discrete emotion detection, offensiveness detection, and moral foundations analysis. The study also highlights GPT's effectiveness in less commonly studied languages, such as African languages, and its ability to detect complex constructs like moral foundations. GPT's performance was compared to other methods, including dictionary-based approaches, and it was found to be significantly more accurate. The study also examined GPT's test-retest reliability and found it to be highly reliable. The researchers conclude that GPT is an effective and accessible tool for multilingual psychological text analysis, capable of facilitating more cross-linguistic research with understudied languages.This study evaluates the effectiveness of GPT, a large language model, in detecting psychological constructs in multilingual text. The researchers tested GPT across 15 datasets containing 47,925 manually annotated tweets and news headlines in 12 languages. They compared GPT's performance with English-language dictionary analysis and several top-performing fine-tuned machine learning models. GPT outperformed dictionary analysis and sometimes matched or exceeded the performance of fine-tuned models. GPT's performance improved across successive versions, particularly for lesser-spoken languages, and became less expensive to use. The study argues that GPT is a superior tool for automated text analysis due to its high accuracy across many languages, lack of need for training data, and ease of use with simple prompts. GPT can be used for tasks such as sentiment analysis, discrete emotion detection, offensiveness detection, and moral foundations analysis. The study also highlights GPT's effectiveness in less commonly studied languages, such as African languages, and its ability to detect complex constructs like moral foundations. GPT's performance was compared to other methods, including dictionary-based approaches, and it was found to be significantly more accurate. The study also examined GPT's test-retest reliability and found it to be highly reliable. The researchers conclude that GPT is an effective and accessible tool for multilingual psychological text analysis, capable of facilitating more cross-linguistic research with understudied languages.