A large annotated corpus for learning natural language inference

A large annotated corpus for learning natural language inference

21 Aug 2015 | Samuel R. Bowman, Gabor Angeli, Christopher Potts, Christopher D. Manning
The Stanford Natural Language Inference (SNLI) corpus is a large, freely available collection of labeled sentence pairs for natural language inference (NLI) tasks. It contains 570,152 sentence pairs, making it two orders of magnitude larger than other NLI resources. The sentences and labels were created by humans in a grounded, naturalistic context, ensuring high quality and reducing annotation errors. The corpus includes three types of labels: entailment, contradiction, and semantic independence. To ensure reliability, four additional judgments were collected for 56,941 examples, with 98% of cases showing consensus among three annotators and 58% showing unanimous agreement among five. The SNLI corpus was created using a crowdsourcing framework, with sentences grounded in specific scenarios and annotated by humans. The corpus includes sentence pairs from the Flickr30k image caption corpus, with each premise sentence describing a scene and the hypothesis sentence being generated by annotators. The corpus is available under a Creative Commons license and includes parses generated by the Stanford PCFG Parser. The paper evaluates various models for NLI, including rule-based systems, simple linear classifiers, and neural network-based models. It finds that a feature-rich classifier and a neural network model based on a Long Short-Term Memory (LSTM) network perform well. The LSTM model is further evaluated using transfer learning, showing competitive performance on the SICK entailment task. The results suggest that the SNLI corpus is a valuable resource for training and evaluating models of semantic representation. The paper also discusses the challenges of NLI, including the difficulty of distinguishing between entailment, contradiction, and semantic independence, and the impact of coreference and event indeterminacy on annotation quality. The SNLI corpus is shown to be a high-quality, large-scale resource that can be used to train and evaluate models for NLI, providing a realistic and challenging testbed for machine learning approaches to semantic representation.The Stanford Natural Language Inference (SNLI) corpus is a large, freely available collection of labeled sentence pairs for natural language inference (NLI) tasks. It contains 570,152 sentence pairs, making it two orders of magnitude larger than other NLI resources. The sentences and labels were created by humans in a grounded, naturalistic context, ensuring high quality and reducing annotation errors. The corpus includes three types of labels: entailment, contradiction, and semantic independence. To ensure reliability, four additional judgments were collected for 56,941 examples, with 98% of cases showing consensus among three annotators and 58% showing unanimous agreement among five. The SNLI corpus was created using a crowdsourcing framework, with sentences grounded in specific scenarios and annotated by humans. The corpus includes sentence pairs from the Flickr30k image caption corpus, with each premise sentence describing a scene and the hypothesis sentence being generated by annotators. The corpus is available under a Creative Commons license and includes parses generated by the Stanford PCFG Parser. The paper evaluates various models for NLI, including rule-based systems, simple linear classifiers, and neural network-based models. It finds that a feature-rich classifier and a neural network model based on a Long Short-Term Memory (LSTM) network perform well. The LSTM model is further evaluated using transfer learning, showing competitive performance on the SICK entailment task. The results suggest that the SNLI corpus is a valuable resource for training and evaluating models of semantic representation. The paper also discusses the challenges of NLI, including the difficulty of distinguishing between entailment, contradiction, and semantic independence, and the impact of coreference and event indeterminacy on annotation quality. The SNLI corpus is shown to be a high-quality, large-scale resource that can be used to train and evaluate models for NLI, providing a realistic and challenging testbed for machine learning approaches to semantic representation.
Reach us at info@study.space
[slides and audio] A large annotated corpus for learning natural language inference