This paper investigates the ability of artificial neural networks (ANNs) to judge the grammatical acceptability of sentences, aiming to test their linguistic competence. The authors introduce the Corpus of Linguistic Acceptability (CoLA), a dataset of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. They train several recurrent neural network models on this dataset and find that their models outperform unsupervised models by Lau et al. (2016) on CoLA. However, all models perform far below human level on a wide range of grammatical constructions. The paper also analyzes the impact of supervised training on acceptability classifiers by varying the domain and quantity of training data, and assesses the models' performance on specific linguistic phenomena. The results suggest that while ANNs can acquire substantial knowledge of grammar, their linguistic competence is far from rivaling humans'. The paper contributes to the growing effort to evaluate ANN models' ability to make fine-grained grammatical distinctions and addresses foundational questions in theoretical linguistics.This paper investigates the ability of artificial neural networks (ANNs) to judge the grammatical acceptability of sentences, aiming to test their linguistic competence. The authors introduce the Corpus of Linguistic Acceptability (CoLA), a dataset of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. They train several recurrent neural network models on this dataset and find that their models outperform unsupervised models by Lau et al. (2016) on CoLA. However, all models perform far below human level on a wide range of grammatical constructions. The paper also analyzes the impact of supervised training on acceptability classifiers by varying the domain and quantity of training data, and assesses the models' performance on specific linguistic phenomena. The results suggest that while ANNs can acquire substantial knowledge of grammar, their linguistic competence is far from rivaling humans'. The paper contributes to the growing effort to evaluate ANN models' ability to make fine-grained grammatical distinctions and addresses foundational questions in theoretical linguistics.