SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languages

SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languages

17 Apr 2024 | Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said Ahmad, Sanchit Ahuja, Alham Fikri Aji, Vladimir Araujo, Meriem Beloucif, Christine De Kock, Oumaima Hourrane, Manish Shrivastava, Thamar Solorio, Nirmal Surange, Krishnapriya Vishnubhotla, Seid Muhie Yimam, Saif M. Mohammad
The SemEval-2024 Task 1 focused on semantic textual relatedness (STR) across 14 languages: Afrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi, Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic, Punjabi, Spanish, and Telugu. These languages belong to five distinct families and are predominantly spoken in Africa and Asia, regions with limited NLP resources. The task aimed to evaluate systems' ability to rank sentence pairs based on their semantic relatedness in three tracks: supervised, unsupervised, and cross-lingual. A total of 163 participants submitted 70 systems across 51 teams, with 38 system description papers. The official evaluation metric was the Spearman rank correlation coefficient, measuring alignment between system predictions and human judgments. The task provided 14 monolingual STR datasets, each containing sentence pairs with relatedness scores between 0 and 1. Data was collected using various methods, including lexical overlap, paraphrases, and BWS annotations. The best-performing systems in Track A (supervised) included AAdaM, which used data augmentation and fine-tuning, and PEAR, which experimented with multilingual embeddings. In Track B (unsupervised), SATLab used character n-grams, while MasonTigers employed embeddings and statistical models. In Track C (cross-lingual), AAdaM and UAlberta used fine-tuning and cross-lingual transfer. The results showed that performance varied across languages, with some low-resource languages outperforming high-resource ones. The task highlighted the importance of language-specific features and cross-lingual approaches. Overall, the task demonstrated the challenges of semantic textual relatedness in low-resource languages and the effectiveness of various machine learning techniques. The datasets and methods presented provide a valuable resource for further research in NLP.The SemEval-2024 Task 1 focused on semantic textual relatedness (STR) across 14 languages: Afrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi, Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic, Punjabi, Spanish, and Telugu. These languages belong to five distinct families and are predominantly spoken in Africa and Asia, regions with limited NLP resources. The task aimed to evaluate systems' ability to rank sentence pairs based on their semantic relatedness in three tracks: supervised, unsupervised, and cross-lingual. A total of 163 participants submitted 70 systems across 51 teams, with 38 system description papers. The official evaluation metric was the Spearman rank correlation coefficient, measuring alignment between system predictions and human judgments. The task provided 14 monolingual STR datasets, each containing sentence pairs with relatedness scores between 0 and 1. Data was collected using various methods, including lexical overlap, paraphrases, and BWS annotations. The best-performing systems in Track A (supervised) included AAdaM, which used data augmentation and fine-tuning, and PEAR, which experimented with multilingual embeddings. In Track B (unsupervised), SATLab used character n-grams, while MasonTigers employed embeddings and statistical models. In Track C (cross-lingual), AAdaM and UAlberta used fine-tuning and cross-lingual transfer. The results showed that performance varied across languages, with some low-resource languages outperforming high-resource ones. The task highlighted the importance of language-specific features and cross-lingual approaches. Overall, the task demonstrated the challenges of semantic textual relatedness in low-resource languages and the effectiveness of various machine learning techniques. The datasets and methods presented provide a valuable resource for further research in NLP.
Reach us at info@study.space
[slides] SemEval Task 1%3A Semantic Textual Relatedness for African and Asian Languages | StudySpace