This paper introduces an adversarial evaluation method for reading comprehension systems, specifically for the Stanford Question Answering Dataset (SQuAD). The goal is to assess whether systems truly understand language, rather than just memorizing patterns. The method involves adding adversarial sentences to paragraphs, which are designed to confuse models without changing the correct answer or misleading humans. The results show that the accuracy of sixteen published models drops significantly when exposed to these adversarial examples, from an average of 75% F1 score to 36%. When the adversary adds ungrammatical sequences, the average accuracy on four models drops further to 7%. The authors argue that this highlights the need for models that understand language more deeply. They also release their code and data publicly to encourage further research. The paper discusses the SQuAD task, the models used, and the adversarial evaluation framework. It also presents experiments showing that no published open-source model is robust to adversarial sentences. The authors also explore the transferability of adversarial examples across models and the effectiveness of training on adversarial examples. The paper concludes that current models are overly stable to semantic changes and that more sophisticated models are needed to understand language at a deeper level.This paper introduces an adversarial evaluation method for reading comprehension systems, specifically for the Stanford Question Answering Dataset (SQuAD). The goal is to assess whether systems truly understand language, rather than just memorizing patterns. The method involves adding adversarial sentences to paragraphs, which are designed to confuse models without changing the correct answer or misleading humans. The results show that the accuracy of sixteen published models drops significantly when exposed to these adversarial examples, from an average of 75% F1 score to 36%. When the adversary adds ungrammatical sequences, the average accuracy on four models drops further to 7%. The authors argue that this highlights the need for models that understand language more deeply. They also release their code and data publicly to encourage further research. The paper discusses the SQuAD task, the models used, and the adversarial evaluation framework. It also presents experiments showing that no published open-source model is robust to adversarial sentences. The authors also explore the transferability of adversarial examples across models and the effectiveness of training on adversarial examples. The paper concludes that current models are overly stable to semantic changes and that more sophisticated models are needed to understand language at a deeper level.