The ArAIEval shared task, part of the ArabicNLP 2024 conference, focuses on detecting propagandistic techniques in Arabic content, both unimodal (text) and multimodal (memes). The task includes two main components: (i) identifying propagandistic textual spans with persuasion techniques in tweets and news articles, and (ii) distinguishing between propagandistic and non-propagandistic memes. A total of 45 teams registered, with 6 and 9 teams submitting systems for Tasks 1 and 2, respectively. Eleven teams also submitted system description papers. Most systems used transformer-based models, such as AraBERT, mBERT, and CAMeLBERT, and applied data augmentation techniques. All systems outperformed random baselines.
Task 1 involves detecting propagandistic techniques in text, with a dataset of 9,000 snippets, including tweets and news paragraphs. The dataset was annotated by three annotators, with majority voting used to determine the final labels. The task required identifying specific persuasion techniques, such as name calling, appeal to fear, and strawman arguments, across different text genres. The evaluation used a modified F1 score to measure performance.
Task 2 focuses on classifying memes as propagandistic or not, with three subtasks: (i) classifying text from memes, (ii) classifying full memes, and (iii) classifying both text and memes. The dataset includes 3,000 memes, with 40% labeled as propagandistic and 60% as non-propagandistic. The task was split into training, development, and test sets, with 70%, 10%, and 20% respectively. The evaluation used macro F1 as the official measure.
The task attracted significant interest, with teams employing various approaches, including deep learning models, transformer-based architectures, and multimodal fusion techniques. The results showed that systems using transformer models, such as MARBERT and CAMeLBERT, performed well. The task highlights the importance of detecting propaganda in Arabic media, with future work focusing on incorporating Arabic dialects and improving model robustness through larger datasets and better handling of class imbalance. Ethical considerations include the potential misuse of models for creating more sophisticated propaganda, emphasizing the need for responsible development and deployment.The ArAIEval shared task, part of the ArabicNLP 2024 conference, focuses on detecting propagandistic techniques in Arabic content, both unimodal (text) and multimodal (memes). The task includes two main components: (i) identifying propagandistic textual spans with persuasion techniques in tweets and news articles, and (ii) distinguishing between propagandistic and non-propagandistic memes. A total of 45 teams registered, with 6 and 9 teams submitting systems for Tasks 1 and 2, respectively. Eleven teams also submitted system description papers. Most systems used transformer-based models, such as AraBERT, mBERT, and CAMeLBERT, and applied data augmentation techniques. All systems outperformed random baselines.
Task 1 involves detecting propagandistic techniques in text, with a dataset of 9,000 snippets, including tweets and news paragraphs. The dataset was annotated by three annotators, with majority voting used to determine the final labels. The task required identifying specific persuasion techniques, such as name calling, appeal to fear, and strawman arguments, across different text genres. The evaluation used a modified F1 score to measure performance.
Task 2 focuses on classifying memes as propagandistic or not, with three subtasks: (i) classifying text from memes, (ii) classifying full memes, and (iii) classifying both text and memes. The dataset includes 3,000 memes, with 40% labeled as propagandistic and 60% as non-propagandistic. The task was split into training, development, and test sets, with 70%, 10%, and 20% respectively. The evaluation used macro F1 as the official measure.
The task attracted significant interest, with teams employing various approaches, including deep learning models, transformer-based architectures, and multimodal fusion techniques. The results showed that systems using transformer models, such as MARBERT and CAMeLBERT, performed well. The task highlights the importance of detecting propaganda in Arabic media, with future work focusing on incorporating Arabic dialects and improving model robustness through larger datasets and better handling of class imbalance. Ethical considerations include the potential misuse of models for creating more sophisticated propaganda, emphasizing the need for responsible development and deployment.