HellaSwag: Can a Machine Really Finish Your Sentence? This paper introduces HellaSwag, a new benchmark for commonsense natural language inference (NLI), which challenges state-of-the-art models to complete sentences in a way that is easy for humans but difficult for machines. The dataset is created using Adversarial Filtering (AF), a process that generates challenging machine-generated answers that are easy for humans to judge as incorrect, yet often misclassified by models. HellaSwag consists of 70,000 examples, with human accuracy exceeding 95%, while machine performance is below 50%. This highlights the difficulty of commonsense reasoning for deep pretrained models, even when they are fine-tuned on the dataset. The paper shows that models like BERT, while achieving high accuracy on the SWAG dataset, struggle with HellaSwag. This is because BERT's performance is heavily dependent on the distributional biases in the training data, and it fails to generalize when the distribution shifts. The dataset is constructed using a combination of state-of-the-art generators and discriminators, and it includes examples from diverse domains such as ActivityNet and WikiHow. The results indicate that HellaSwag is a challenging testbed for NLI models, and that the task of commonsense reasoning remains unsolved. The paper also discusses the importance of evolving benchmarks in NLP research, where datasets co-evolve with the state-of-the-art to present ever-harder challenges. This approach ensures that benchmarks remain relevant and effective in evaluating the capabilities of models. The results show that even with extensive pretraining, models like BERT struggle with HellaSwag, and that the task of commonsense reasoning requires more than just surface-level pattern recognition. The paper concludes that the task of commonsense NLI remains unsolved, and that the development of more sophisticated models and algorithms is necessary to achieve human-level performance. The results also highlight the importance of human validation in dataset creation, and the need for careful curation to ensure that the data is challenging for models but easy for humans to judge. The paper provides a comprehensive analysis of the challenges in commonsense reasoning and suggests a path forward for NLP research.HellaSwag: Can a Machine Really Finish Your Sentence? This paper introduces HellaSwag, a new benchmark for commonsense natural language inference (NLI), which challenges state-of-the-art models to complete sentences in a way that is easy for humans but difficult for machines. The dataset is created using Adversarial Filtering (AF), a process that generates challenging machine-generated answers that are easy for humans to judge as incorrect, yet often misclassified by models. HellaSwag consists of 70,000 examples, with human accuracy exceeding 95%, while machine performance is below 50%. This highlights the difficulty of commonsense reasoning for deep pretrained models, even when they are fine-tuned on the dataset. The paper shows that models like BERT, while achieving high accuracy on the SWAG dataset, struggle with HellaSwag. This is because BERT's performance is heavily dependent on the distributional biases in the training data, and it fails to generalize when the distribution shifts. The dataset is constructed using a combination of state-of-the-art generators and discriminators, and it includes examples from diverse domains such as ActivityNet and WikiHow. The results indicate that HellaSwag is a challenging testbed for NLI models, and that the task of commonsense reasoning remains unsolved. The paper also discusses the importance of evolving benchmarks in NLP research, where datasets co-evolve with the state-of-the-art to present ever-harder challenges. This approach ensures that benchmarks remain relevant and effective in evaluating the capabilities of models. The results show that even with extensive pretraining, models like BERT struggle with HellaSwag, and that the task of commonsense reasoning requires more than just surface-level pattern recognition. The paper concludes that the task of commonsense NLI remains unsolved, and that the development of more sophisticated models and algorithms is necessary to achieve human-level performance. The results also highlight the importance of human validation in dataset creation, and the need for careful curation to ensure that the data is challenging for models but easy for humans to judge. The paper provides a comprehensive analysis of the challenges in commonsense reasoning and suggests a path forward for NLP research.