Semantic distance in WordNet: An experimental, application-oriented evaluation of five measures

Semantic distance in WordNet: An experimental, application-oriented evaluation of five measures

| Alexander Budanitsky and Graeme Hirst
This paper evaluates five measures of semantic relatedness in WordNet by testing their performance in a real-world spelling correction system. The five measures are: Hirst–St-Onge, Leacock–Chodorow, Resnik, Jiang–Conrath, and Lin. The study finds that Jiang and Conrath's measure performs best overall, while Hirst and St-Onge's measure over-relates, Resnik's under-relates, and Lin and Leacock and Chodorow's measures fall in between. The paper discusses the importance of semantic relatedness in NLP applications, such as word sense disambiguation, text summarization, and information retrieval. It also highlights the challenges of evaluating semantic similarity measures, noting that human judgments are ideal but difficult to obtain. The paper proposes an application-based evaluation of the measures using a malapropism detection task, where the goal is to detect and correct spelling errors in open-class words. The evaluation involves testing the five measures on a corpus of 107,233 words, with 1,408 being malapropisms. The measures are tested with different search scopes (window sizes) to assess their performance in detecting and correcting malapropisms. The results show that Jiang–Conrath's measure performs best in both suspicion and detection tasks, followed by Lin and Leacock–Chodorow, then Resnik, and finally Hirst–St-Onge. The study concludes that Jiang and Conrath's measure is the most effective for semantic relatedness in NLP applications. However, it also notes that the performance of the measures is limited by the assumptions made in the malapropism detection task. The paper emphasizes the need for more data and a better understanding of semantic relatedness to improve the accuracy of these measures.This paper evaluates five measures of semantic relatedness in WordNet by testing their performance in a real-world spelling correction system. The five measures are: Hirst–St-Onge, Leacock–Chodorow, Resnik, Jiang–Conrath, and Lin. The study finds that Jiang and Conrath's measure performs best overall, while Hirst and St-Onge's measure over-relates, Resnik's under-relates, and Lin and Leacock and Chodorow's measures fall in between. The paper discusses the importance of semantic relatedness in NLP applications, such as word sense disambiguation, text summarization, and information retrieval. It also highlights the challenges of evaluating semantic similarity measures, noting that human judgments are ideal but difficult to obtain. The paper proposes an application-based evaluation of the measures using a malapropism detection task, where the goal is to detect and correct spelling errors in open-class words. The evaluation involves testing the five measures on a corpus of 107,233 words, with 1,408 being malapropisms. The measures are tested with different search scopes (window sizes) to assess their performance in detecting and correcting malapropisms. The results show that Jiang–Conrath's measure performs best in both suspicion and detection tasks, followed by Lin and Leacock–Chodorow, then Resnik, and finally Hirst–St-Onge. The study concludes that Jiang and Conrath's measure is the most effective for semantic relatedness in NLP applications. However, it also notes that the performance of the measures is limited by the assumptions made in the malapropism detection task. The paper emphasizes the need for more data and a better understanding of semantic relatedness to improve the accuracy of these measures.
Reach us at info@study.space
[slides and audio] Semantic distance in WordNet%3A An experimental%2C application-oriented evaluation of five measures