The paper addresses the issue of large language models (LLMs) being trained on unlicensed data, leading to copyright disputes. It critiques existing membership inference attacks (MIAs) that attempt to identify if specific text sequences were part of the training data. The authors argue that these MIAs often fail due to distribution shifts, such as temporal differences in the data, rather than inherent vulnerabilities. They propose a new method called *dataset inference* to accurately identify the datasets used to train LLMs. This method combines multiple MIAs that provide positive signals for a given distribution and performs a statistical test to distinguish between training and validation sets. The approach is evaluated using the Pythia suite of models trained on the PILE dataset, achieving statistically significant p-values < 0.1 without false positives. The paper also provides guidelines for future research on MIAs, emphasizing the importance of using IID splits, considering various data distributions, and evaluating false positives.The paper addresses the issue of large language models (LLMs) being trained on unlicensed data, leading to copyright disputes. It critiques existing membership inference attacks (MIAs) that attempt to identify if specific text sequences were part of the training data. The authors argue that these MIAs often fail due to distribution shifts, such as temporal differences in the data, rather than inherent vulnerabilities. They propose a new method called *dataset inference* to accurately identify the datasets used to train LLMs. This method combines multiple MIAs that provide positive signals for a given distribution and performs a statistical test to distinguish between training and validation sets. The approach is evaluated using the Pythia suite of models trained on the PILE dataset, achieving statistically significant p-values < 0.1 without false positives. The paper also provides guidelines for future research on MIAs, emphasizing the importance of using IID splits, considering various data distributions, and evaluating false positives.