A large annotated corpus for learning natural language inference

A large annotated corpus for learning natural language inference

21 Aug 2015 | Samuel R. Bowman, Gabor Angeli, Christopher Potts, Christopher D. Manning
The paper introduces the Stanford Natural Language Inference (SNLI) corpus, a large-scale collection of labeled sentence pairs designed to address the limitations of existing resources in the field of natural language inference (NLI). SNLI contains 570,152 pairs of sentences, significantly larger than other resources, and is the first to be fully human-labeled in a grounded, naturalistic context. The corpus aims to provide a high-quality, diverse dataset for training modern, data-intensive models, particularly neural networks, which have struggled with NLI tasks due to the lack of sufficient data. The authors describe the challenges faced by existing NLI resources, such as their small size, algorithmically generated sentences, and issues with event and entity coreference. To address these issues, SNLI employs a crowdsourcing framework that ensures consistent and reliable annotations. The data is collected using Amazon Mechanical Turk, where workers are presented with premise scene descriptions from the Flickr30k corpus and asked to generate hypothesis sentences for three labels: entailment, neutral, and contradiction. The paper evaluates various models on SNLI, including rule-based systems, simple linear classifiers, and neural network-based models. It finds that both lexicalized classifiers and neural network models, particularly those using Long Short-Term Memory (LSTM) networks, perform well. The LSTM model, in particular, shows competitive performance on standard NLI benchmarks and can be adapted to other NLI tasks through transfer learning, achieving the best reported performance by a neural network model. The authors conclude that SNLI provides valuable training data and a challenging testbed for further research in machine learning for semantic representation, highlighting the importance of large-scale, high-quality resources in advancing the field of NLI.The paper introduces the Stanford Natural Language Inference (SNLI) corpus, a large-scale collection of labeled sentence pairs designed to address the limitations of existing resources in the field of natural language inference (NLI). SNLI contains 570,152 pairs of sentences, significantly larger than other resources, and is the first to be fully human-labeled in a grounded, naturalistic context. The corpus aims to provide a high-quality, diverse dataset for training modern, data-intensive models, particularly neural networks, which have struggled with NLI tasks due to the lack of sufficient data. The authors describe the challenges faced by existing NLI resources, such as their small size, algorithmically generated sentences, and issues with event and entity coreference. To address these issues, SNLI employs a crowdsourcing framework that ensures consistent and reliable annotations. The data is collected using Amazon Mechanical Turk, where workers are presented with premise scene descriptions from the Flickr30k corpus and asked to generate hypothesis sentences for three labels: entailment, neutral, and contradiction. The paper evaluates various models on SNLI, including rule-based systems, simple linear classifiers, and neural network-based models. It finds that both lexicalized classifiers and neural network models, particularly those using Long Short-Term Memory (LSTM) networks, perform well. The LSTM model, in particular, shows competitive performance on standard NLI benchmarks and can be adapted to other NLI tasks through transfer learning, achieving the best reported performance by a neural network model. The authors conclude that SNLI provides valuable training data and a challenging testbed for further research in machine learning for semantic representation, highlighting the importance of large-scale, high-quality resources in advancing the field of NLI.
Reach us at info@study.space