ArAIEval Shared Task: Propagandistic Techniques Detection in Unimodal and Multimodal Arabic Content

ArAIEval Shared Task: Propagandistic Techniques Detection in Unimodal and Multimodal Arabic Content

5 Jul 2024 | Maram Hasanain, Md. Arid Hasan, Fatema Ahmed, Reem Suwaileh, Md. Raful Biswas, Wajdi Zaghouani, Firoj Alam
The paper presents the second edition of the ArAIEval shared task, organized as part of the ArabicNLP 2024 conference. The task focuses on two main objectives: (i) detecting propagandistic textual spans with persuasion techniques identification in tweets and news articles, and (ii) distinguishing between propagandistic and non-propagandistic memes. A total of 14 teams participated in the final evaluation phase, with 6 and 9 teams participating in Tasks 1 and 2, respectively. All systems utilized fine-tuned transformer models, such as AraBERT, and several applied data augmentation techniques. The datasets and evaluation scripts are released to the research community to enable further research on these important tasks in Arabic. Online media has become a primary channel for information dissemination, but it is also a platform for malicious actors to spread misinformation and propaganda. Propaganda is a form of communication designed to influence opinions and actions using rhetorical and psychological devices. The task aims to develop systems that can detect specific propagandistic techniques within textual content and classify propagandistic memes. The task is divided into two main parts: (i) detecting propagandistic techniques in textual content, and (ii) classifying propagandistic memes. The textual content includes tweets and news articles, while the memes dataset consists of approximately 3,000 annotated memes collected from various social media platforms. The evaluation setup includes a development phase and a test phase, with systems evaluated using modified F1 measures for multilabel sequence tagging tasks. The datasets are constructed from real-world sources, with careful annotation processes to ensure accuracy. A total of 14 teams participated in the final evaluation, with 6 and 9 teams submitting runs for Tasks 1 and 2, respectively. The majority of systems used fine-tuned transformer models, and several applied data augmentation techniques. All systems outperformed random baselines, with notable performances from teams using AraBERT and other transformer models. The paper discusses the limitations of the datasets, which are skewed in label distribution, and suggests future work to address this issue. Ethical considerations are also discussed, emphasizing the importance of unbiased and responsible use of the developed models. The ArAIEval shared task has successfully engaged the research community in Arabic AI, particularly in the area of propaganda detection. The release of datasets and evaluation scripts will facilitate further research and development in this critical domain.The paper presents the second edition of the ArAIEval shared task, organized as part of the ArabicNLP 2024 conference. The task focuses on two main objectives: (i) detecting propagandistic textual spans with persuasion techniques identification in tweets and news articles, and (ii) distinguishing between propagandistic and non-propagandistic memes. A total of 14 teams participated in the final evaluation phase, with 6 and 9 teams participating in Tasks 1 and 2, respectively. All systems utilized fine-tuned transformer models, such as AraBERT, and several applied data augmentation techniques. The datasets and evaluation scripts are released to the research community to enable further research on these important tasks in Arabic. Online media has become a primary channel for information dissemination, but it is also a platform for malicious actors to spread misinformation and propaganda. Propaganda is a form of communication designed to influence opinions and actions using rhetorical and psychological devices. The task aims to develop systems that can detect specific propagandistic techniques within textual content and classify propagandistic memes. The task is divided into two main parts: (i) detecting propagandistic techniques in textual content, and (ii) classifying propagandistic memes. The textual content includes tweets and news articles, while the memes dataset consists of approximately 3,000 annotated memes collected from various social media platforms. The evaluation setup includes a development phase and a test phase, with systems evaluated using modified F1 measures for multilabel sequence tagging tasks. The datasets are constructed from real-world sources, with careful annotation processes to ensure accuracy. A total of 14 teams participated in the final evaluation, with 6 and 9 teams submitting runs for Tasks 1 and 2, respectively. The majority of systems used fine-tuned transformer models, and several applied data augmentation techniques. All systems outperformed random baselines, with notable performances from teams using AraBERT and other transformer models. The paper discusses the limitations of the datasets, which are skewed in label distribution, and suggests future work to address this issue. Ethical considerations are also discussed, emphasizing the importance of unbiased and responsible use of the developed models. The ArAIEval shared task has successfully engaged the research community in Arabic AI, particularly in the area of propaganda detection. The release of datasets and evaluation scripts will facilitate further research and development in this critical domain.
Reach us at info@study.space
[slides] ArAIEval Shared Task%3A Propagandistic Techniques Detection in Unimodal and Multimodal Arabic Content | StudySpace