ChatGPT outperforms crowd workers for text-annotation tasks

ChatGPT outperforms crowd workers for text-annotation tasks

July 18, 2023 | Fabrizio Gilardi, Meysam Alizadeh, Maël Kubli
The paper demonstrates that ChatGPT outperforms crowd workers in text-annotation tasks, particularly for relevance, stance, topics, and frame detection. Using four datasets totaling 6,183 tweets and news articles, the study found that ChatGPT's zero-shot accuracy was about 25 percentage points higher than that of crowd workers on MTurk, with intercoder agreement also superior. Additionally, ChatGPT's per-annotation cost was significantly lower, at approximately $0.003, making it a much more cost-effective solution. The results suggest that large language models like ChatGPT have the potential to revolutionize text annotation, potentially transforming how researchers conduct data annotations and disrupting the business models of platforms like MTurk. Further research is needed to explore the capabilities of LLMs in broader contexts, including multiple languages and different types of tasks.The paper demonstrates that ChatGPT outperforms crowd workers in text-annotation tasks, particularly for relevance, stance, topics, and frame detection. Using four datasets totaling 6,183 tweets and news articles, the study found that ChatGPT's zero-shot accuracy was about 25 percentage points higher than that of crowd workers on MTurk, with intercoder agreement also superior. Additionally, ChatGPT's per-annotation cost was significantly lower, at approximately $0.003, making it a much more cost-effective solution. The results suggest that large language models like ChatGPT have the potential to revolutionize text annotation, potentially transforming how researchers conduct data annotations and disrupting the business models of platforms like MTurk. Further research is needed to explore the capabilities of LLMs in broader contexts, including multiple languages and different types of tasks.
Reach us at info@study.space
[slides] ChatGPT outperforms crowd workers for text-annotation tasks | StudySpace