This paper investigates the use of copyright traps to detect the use of copyrighted materials in large language models (LLMs), particularly for models that do not naturally memorize training data. The authors propose injecting fictitious text sequences (trap sequences) into original content to enable document-level membership inference. They train a 1.3B parameter LLM from scratch on a large dataset of public domain books and evaluate the detectability of trap sequences under various conditions. The study shows that existing methods for detecting memorization in LLMs are ineffective against models that do not naturally memorize, such as the 1.3B LLM used in this experiment. However, longer sequences repeated many times can be reliably detected, achieving an AUC of 0.75. The authors also find that sequences with higher perplexity (as measured by a reference model) are more detectable, suggesting that perplexity could be a confounding factor in prior studies of LLM memorization. The research contributes to the understanding of LLM memorization by enabling causal relationships between memorization and sequence properties. The authors also show that the inclusion of trap sequences in training data can be effective for detecting the use of copyrighted materials, even in models that would not naturally memorize sufficient content for such detection. The study highlights the importance of randomized controlled experiments in understanding LLM memorization and the potential for copyright traps to be used as a tool for detecting the use of copyrighted materials in LLM training. The authors also discuss the practical implications of their findings, including the impact on content readability and the potential for misuse of LLMs for producing misinformation. The paper concludes with a call for further research into the design of trap sequences that maximize detectability and the development of effective defenses against potential misuse of LLMs.This paper investigates the use of copyright traps to detect the use of copyrighted materials in large language models (LLMs), particularly for models that do not naturally memorize training data. The authors propose injecting fictitious text sequences (trap sequences) into original content to enable document-level membership inference. They train a 1.3B parameter LLM from scratch on a large dataset of public domain books and evaluate the detectability of trap sequences under various conditions. The study shows that existing methods for detecting memorization in LLMs are ineffective against models that do not naturally memorize, such as the 1.3B LLM used in this experiment. However, longer sequences repeated many times can be reliably detected, achieving an AUC of 0.75. The authors also find that sequences with higher perplexity (as measured by a reference model) are more detectable, suggesting that perplexity could be a confounding factor in prior studies of LLM memorization. The research contributes to the understanding of LLM memorization by enabling causal relationships between memorization and sequence properties. The authors also show that the inclusion of trap sequences in training data can be effective for detecting the use of copyrighted materials, even in models that would not naturally memorize sufficient content for such detection. The study highlights the importance of randomized controlled experiments in understanding LLM memorization and the potential for copyright traps to be used as a tool for detecting the use of copyrighted materials in LLM training. The authors also discuss the practical implications of their findings, including the impact on content readability and the potential for misuse of LLMs for producing misinformation. The paper concludes with a call for further research into the design of trap sequences that maximize detectability and the development of effective defenses against potential misuse of LLMs.