SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generation Large Language Models (LLMs) have transformed machine learning but raise significant legal concerns due to their potential to generate text that infringes on copyrights, leading to high-profile lawsuits. The legal landscape struggles to keep pace with these advancements, with ongoing debates about whether generated text might plagiarize copyrighted materials. Current LLMs may infringe on copyrights or overly restrict non-copyrighted texts, resulting in three main challenges: (i) the need for a comprehensive evaluation benchmark to assess copyright compliance from multiple aspects; (ii) evaluating robustness against safeguard bypassing attacks; and (iii) developing effective defenses targeted against the generation of copyrighted text. To address these challenges, we introduce a curated dataset to evaluate methods, test attack strategies, and propose a lightweight, real-time defense mechanism to prevent the generation of copyrighted text, ensuring the safe and lawful use of LLMs. Our experiments show that current LLMs frequently output copyrighted text, and jailbreaking attacks can significantly increase the volume of copyrighted output. Our proposed defense mechanism significantly reduces the volume of copyrighted text generated by LLMs by effectively refusing malicious requests. We construct a meticulously curated dataset of (i) copyrighted text; (ii) non-copyrighted text; and (iii) text with varying copyright status across different countries. This dataset is manually evaluated to ensure correct labeling. We also evaluate the robustness of LLMs by adopting jailbreaking attacks and introduce the rate of refusal, a common evaluation metric in the jailbreaking field, in our evaluation protocol. Our findings indicate that these attacks can lead to an increased volume of copyrighted text being generated by LLMs, suggesting that current LLMs remain vulnerable to requests for copyrighted material, motivating the need to develop defense mechanisms focused on copyright protection. Although various methods may be used to prevent LLMs from generating copyrighted text, they all have limitations. We propose an easy-to-deploy, agent-based defense mechanism that prevents any LLM from generating copyrighted text by checking real-time information from web searches. Our approach involves recognizing and remembering copyrighted content, letting the LLM clearly reject the request when copyrighted text is relevant. Our defense mechanism does not interfere when no copyrighted text is relevant to the request. In this work, we integrate the benchmark, robustness, and defense method as a comprehensive framework, namely SHIELD, standing for System for Handling Intellectual Property and Evaluation of LLM-Generated Text for Legal Defense. Our contributions are summarized as follows: • We construct a meticulously curated dataset of copyrighted and non-copyrighted text to evaluate various approaches. The dataset is manually reviewed to ensure accurate labeling. • We are the first to evaluate defense mechanisms against jailbreaking attacks generating copyrighted text. We show that safeguards on copyright compliance can be bypassed by malicious users with simple prompt engineering. • We propose novel agent-based defense to prevent LLMs from generating copyrighted text, which best protects intellectual propertySHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generation Large Language Models (LLMs) have transformed machine learning but raise significant legal concerns due to their potential to generate text that infringes on copyrights, leading to high-profile lawsuits. The legal landscape struggles to keep pace with these advancements, with ongoing debates about whether generated text might plagiarize copyrighted materials. Current LLMs may infringe on copyrights or overly restrict non-copyrighted texts, resulting in three main challenges: (i) the need for a comprehensive evaluation benchmark to assess copyright compliance from multiple aspects; (ii) evaluating robustness against safeguard bypassing attacks; and (iii) developing effective defenses targeted against the generation of copyrighted text. To address these challenges, we introduce a curated dataset to evaluate methods, test attack strategies, and propose a lightweight, real-time defense mechanism to prevent the generation of copyrighted text, ensuring the safe and lawful use of LLMs. Our experiments show that current LLMs frequently output copyrighted text, and jailbreaking attacks can significantly increase the volume of copyrighted output. Our proposed defense mechanism significantly reduces the volume of copyrighted text generated by LLMs by effectively refusing malicious requests. We construct a meticulously curated dataset of (i) copyrighted text; (ii) non-copyrighted text; and (iii) text with varying copyright status across different countries. This dataset is manually evaluated to ensure correct labeling. We also evaluate the robustness of LLMs by adopting jailbreaking attacks and introduce the rate of refusal, a common evaluation metric in the jailbreaking field, in our evaluation protocol. Our findings indicate that these attacks can lead to an increased volume of copyrighted text being generated by LLMs, suggesting that current LLMs remain vulnerable to requests for copyrighted material, motivating the need to develop defense mechanisms focused on copyright protection. Although various methods may be used to prevent LLMs from generating copyrighted text, they all have limitations. We propose an easy-to-deploy, agent-based defense mechanism that prevents any LLM from generating copyrighted text by checking real-time information from web searches. Our approach involves recognizing and remembering copyrighted content, letting the LLM clearly reject the request when copyrighted text is relevant. Our defense mechanism does not interfere when no copyrighted text is relevant to the request. In this work, we integrate the benchmark, robustness, and defense method as a comprehensive framework, namely SHIELD, standing for System for Handling Intellectual Property and Evaluation of LLM-Generated Text for Legal Defense. Our contributions are summarized as follows: • We construct a meticulously curated dataset of copyrighted and non-copyrighted text to evaluate various approaches. The dataset is manually reviewed to ensure accurate labeling. • We are the first to evaluate defense mechanisms against jailbreaking attacks generating copyrighted text. We show that safeguards on copyright compliance can be bypassed by malicious users with simple prompt engineering. • We propose novel agent-based defense to prevent LLMs from generating copyrighted text, which best protects intellectual property