The FIGNEWS Shared Task on News Media Narratives

The FIGNEWS Shared Task on News Media Narratives

25 Jul 2024 | Wajdi Zaghouani, Mustafa Jarrar, Nizar Habash, Houda Bouamor, Imed Zitouni, Mona Diab, Samhaa R. El-Beltagy, Muhammad AbuOdeh
The FIGNEWS shared task, organized as part of the Arabic-NLP 2024 conference, aimed to address bias and propaganda in multilingual news posts, focusing on the early days of the Israel War on Gaza. The task involved 17 teams across five languages (English, French, Arabic, Hebrew, Hindi) and produced 129,800 data points through four evaluation tracks: guidelines development, annotation quality, quantity, and consistency. Teams annotated posts to classify them as unbiased, biased against Palestine, Israel, both, others, or unclear, and as propaganda or not. Key findings included high variability in inter-annotator agreement, with within-team scores significantly higher than across-team ones. The task highlighted the challenges of bias and propaganda detection, emphasizing the need for clear guidelines and collaborative efforts. The shared task contributed to the development of annotation frameworks and datasets for future research in NLP and media analysis. The results showed that teams like NLPColab, Sina, and The Lexicon Ladies performed well in various tracks. The task also addressed ethical considerations, ensuring data anonymity and responsible use. Limitations included subjective annotation, limited label diversity, and potential sampling bias. The FIGNEWS shared task represents a significant step forward in data annotation and media analysis, promoting interdisciplinary research and improving media literacy.The FIGNEWS shared task, organized as part of the Arabic-NLP 2024 conference, aimed to address bias and propaganda in multilingual news posts, focusing on the early days of the Israel War on Gaza. The task involved 17 teams across five languages (English, French, Arabic, Hebrew, Hindi) and produced 129,800 data points through four evaluation tracks: guidelines development, annotation quality, quantity, and consistency. Teams annotated posts to classify them as unbiased, biased against Palestine, Israel, both, others, or unclear, and as propaganda or not. Key findings included high variability in inter-annotator agreement, with within-team scores significantly higher than across-team ones. The task highlighted the challenges of bias and propaganda detection, emphasizing the need for clear guidelines and collaborative efforts. The shared task contributed to the development of annotation frameworks and datasets for future research in NLP and media analysis. The results showed that teams like NLPColab, Sina, and The Lexicon Ladies performed well in various tracks. The task also addressed ethical considerations, ensuring data anonymity and responsible use. Limitations included subjective annotation, limited label diversity, and potential sampling bias. The FIGNEWS shared task represents a significant step forward in data annotation and media analysis, promoting interdisciplinary research and improving media literacy.
Reach us at info@study.space