The paper "Mission: Impossible Language Models" by Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts explores the capabilities of large language models (LLMs) in learning both possible and impossible human languages. The authors develop a set of synthetic "impossible languages" of varying complexity, designed by systematically altering English data with unnatural word orders and grammar rules. These languages range from inherently impossible languages, such as random and irreversible shuffles of English words, to languages that are considered impossible in linguistics but may not be intuitively so. The study uses GPT-2 small models to assess the capacity of these models to learn the impossible languages. The evaluation includes three core experiments: 1. **Experiment 1**: Measures the learning efficiency of GPT-2 models trained on possible and impossible languages through test set perplexities. Results show that models trained on possible languages learn more efficiently. 2. **Experiment 2**: Focuses on count-based verb marking rules in *HOP languages* using surprisal comparisons. GPT-2 trained on possible languages are more surprised by ungrammatical constructions, indicating a preference for natural grammatical structures. 3. **Experiment 3**: Conducts causal abstraction analysis to understand the internal mechanisms of GPT-2 models in learning count-based grammar rules. The results show that models develop natural, modular solutions to unnatural grammatical patterns. The findings challenge the claims made by Chomsky and others that LLMs cannot distinguish between possible and impossible languages. The authors argue that while LLMs struggle with impossible languages, they still provide valuable insights into the cognitive and typological aspects of language learning. The paper opens up a new line of inquiry into how different LLM architectures can be tested on a variety of impossible languages to further advance the understanding of LLMs' capabilities and limitations.The paper "Mission: Impossible Language Models" by Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts explores the capabilities of large language models (LLMs) in learning both possible and impossible human languages. The authors develop a set of synthetic "impossible languages" of varying complexity, designed by systematically altering English data with unnatural word orders and grammar rules. These languages range from inherently impossible languages, such as random and irreversible shuffles of English words, to languages that are considered impossible in linguistics but may not be intuitively so. The study uses GPT-2 small models to assess the capacity of these models to learn the impossible languages. The evaluation includes three core experiments: 1. **Experiment 1**: Measures the learning efficiency of GPT-2 models trained on possible and impossible languages through test set perplexities. Results show that models trained on possible languages learn more efficiently. 2. **Experiment 2**: Focuses on count-based verb marking rules in *HOP languages* using surprisal comparisons. GPT-2 trained on possible languages are more surprised by ungrammatical constructions, indicating a preference for natural grammatical structures. 3. **Experiment 3**: Conducts causal abstraction analysis to understand the internal mechanisms of GPT-2 models in learning count-based grammar rules. The results show that models develop natural, modular solutions to unnatural grammatical patterns. The findings challenge the claims made by Chomsky and others that LLMs cannot distinguish between possible and impossible languages. The authors argue that while LLMs struggle with impossible languages, they still provide valuable insights into the cognitive and typological aspects of language learning. The paper opens up a new line of inquiry into how different LLM architectures can be tested on a variety of impossible languages to further advance the understanding of LLMs' capabilities and limitations.