Crowdsourcing a Word–Emotion Association Lexicon

Crowdsourcing a Word–Emotion Association Lexicon

28 Aug 2013 | SAIF M. MOHAMMAD AND PETER D. TURNEY
This paper presents a method for generating a large, high-quality word-emotion association lexicon using crowdsourcing. The authors show how the collective wisdom of the crowd can be leveraged to create a comprehensive emotion lexicon quickly and inexpensively. They address challenges in emotion annotation within a crowdsourcing context and propose solutions to ensure high-quality annotations. A key innovation is the inclusion of a word choice question to discourage malicious data entry, identify unfamiliar terms, and obtain sense-level annotations. The authors conducted experiments to determine the best way to phrase emotion-annotation questions, finding that asking if a term is associated with an emotion leads to higher inter-annotator agreement than asking if a term evokes an emotion. The resulting EmoLex lexicon contains over 10,000 terms and includes words from various sources, including the General Inquirer and the WordNet Affect Lexicon. The lexicon covers eight basic emotions: joy, sadness, anger, fear, trust, disgust, surprise, and anticipation. The authors analyze the annotations to answer questions about the difficulty of human annotation, the phrasing of emotion-annotation questions, and the agreement among annotators. They also examine whether emotions are more commonly evoked by certain parts of speech and whether there is a correlation between word polarity and emotion. The paper discusses the applications of emotion-aware systems, including customer relationship management, sentiment tracking, and developing intelligent tutoring systems. It also highlights the importance of emotion analysis in understanding how people communicate and in detecting how people use emotion-bearing words to persuade others. The authors also discuss the challenges of emotion annotation in a crowdsourcing setting, including ensuring the quality of annotations and handling ambiguous terms. The authors developed a crowdsourcing approach using Amazon's Mechanical Turk to collect annotations for a large set of terms. They used a word choice question to ensure that annotators were familiar with the target terms and to guide them to the correct sense of the word. The annotations were validated and post-processed to ensure quality. The resulting lexicon contains over 38,000 assignments from approximately 2,216 Turkers. The authors also analyze the agreement among annotators and show that for over 60% of the terms, at least four annotators agreed on the emotion associations. The paper concludes that the crowdsourcing approach is effective for generating a large, high-quality emotion lexicon.This paper presents a method for generating a large, high-quality word-emotion association lexicon using crowdsourcing. The authors show how the collective wisdom of the crowd can be leveraged to create a comprehensive emotion lexicon quickly and inexpensively. They address challenges in emotion annotation within a crowdsourcing context and propose solutions to ensure high-quality annotations. A key innovation is the inclusion of a word choice question to discourage malicious data entry, identify unfamiliar terms, and obtain sense-level annotations. The authors conducted experiments to determine the best way to phrase emotion-annotation questions, finding that asking if a term is associated with an emotion leads to higher inter-annotator agreement than asking if a term evokes an emotion. The resulting EmoLex lexicon contains over 10,000 terms and includes words from various sources, including the General Inquirer and the WordNet Affect Lexicon. The lexicon covers eight basic emotions: joy, sadness, anger, fear, trust, disgust, surprise, and anticipation. The authors analyze the annotations to answer questions about the difficulty of human annotation, the phrasing of emotion-annotation questions, and the agreement among annotators. They also examine whether emotions are more commonly evoked by certain parts of speech and whether there is a correlation between word polarity and emotion. The paper discusses the applications of emotion-aware systems, including customer relationship management, sentiment tracking, and developing intelligent tutoring systems. It also highlights the importance of emotion analysis in understanding how people communicate and in detecting how people use emotion-bearing words to persuade others. The authors also discuss the challenges of emotion annotation in a crowdsourcing setting, including ensuring the quality of annotations and handling ambiguous terms. The authors developed a crowdsourcing approach using Amazon's Mechanical Turk to collect annotations for a large set of terms. They used a word choice question to ensure that annotators were familiar with the target terms and to guide them to the correct sense of the word. The annotations were validated and post-processed to ensure quality. The resulting lexicon contains over 38,000 assignments from approximately 2,216 Turkers. The authors also analyze the agreement among annotators and show that for over 60% of the terms, at least four annotators agreed on the emotion associations. The paper concludes that the crowdsourcing approach is effective for generating a large, high-quality emotion lexicon.
Reach us at info@study.space
Understanding CROWDSOURCING A WORD%E2%80%93EMOTION ASSOCIATION LEXICON