**WAVES (Watermark Analysis via Enhanced Stress-testing)** is a benchmark designed to assess the robustness of image watermarks, addressing the limitations of current evaluation methods. WAVES integrates detection and identification tasks and establishes a standardized evaluation protocol that includes a diverse range of stress tests, from traditional image distortions to advanced, novel variations of diffusive and adversarial attacks. The evaluation focuses on two key dimensions: the degree of image quality degradation and the efficacy of watermark detection after attacks. The project reveals previously undetected vulnerabilities in several modern watermarking algorithms and is intended to serve as a toolkit for future development of robust watermarks. **Key Contributions:** 1. **Comprehensive Evaluation:** WAVES provides a comprehensive evaluation framework that includes 26 diverse attacks across three categories (distortions, image regeneration, and adversarial attacks) and employs 8 quality metrics. 2. **Standardized Metrics:** WAVES prioritizes the True Positive Rate (TPR) at a stringent False Positive Rate (FPR) threshold of 0.1%, addressing the inadequacies of alternative metrics like p-values and AUROC. 3. **Performance vs. Quality Plots:** WAVES introduces Performance vs. Quality 2D plots to comprehensively compare watermark performance and image quality, providing a unified perspective on robustness. 4. **Attack Effectiveness:** WAVES ranks attacks based on their impact on detection performance and image quality, offering insights into the effectiveness of different attacks against various watermarking methods. **Evaluation Results:** - **Watermark Robustness:** StegaStamp demonstrates exceptional robustness, followed by Tree-Ring, while Stable Signature shows the least robustness. - **Attack Effectiveness:** Different watermarking methods exhibit varying vulnerabilities to different types of attacks. For example, Tree-Ring is particularly vulnerable to adversarial attacks, while Stable Signature is susceptible to most regeneration attacks. - **User Identification:** Similar trends in watermark robustness and attack effectiveness are observed in user identification tasks, with watermarks becoming more vulnerable as the number of users increases. **Limitations and Future Directions:** - **Public VAEs:** WAVES highlights the risks of using publicly available VAEs in watermarked diffusion models, as these can be easily compromised by adversarial attacks. - **Future Strategies:** Potential strategies to improve robustness include incorporating more types of transformations, improving algorithmic frameworks, and using redundant bits or hybrid approaches. **Impact Statement:** WAVES contributes to the development of more robust watermarking systems, which are crucial for protecting creative ownership and preventing the misrepresentation of AI-generated content.**WAVES (Watermark Analysis via Enhanced Stress-testing)** is a benchmark designed to assess the robustness of image watermarks, addressing the limitations of current evaluation methods. WAVES integrates detection and identification tasks and establishes a standardized evaluation protocol that includes a diverse range of stress tests, from traditional image distortions to advanced, novel variations of diffusive and adversarial attacks. The evaluation focuses on two key dimensions: the degree of image quality degradation and the efficacy of watermark detection after attacks. The project reveals previously undetected vulnerabilities in several modern watermarking algorithms and is intended to serve as a toolkit for future development of robust watermarks. **Key Contributions:** 1. **Comprehensive Evaluation:** WAVES provides a comprehensive evaluation framework that includes 26 diverse attacks across three categories (distortions, image regeneration, and adversarial attacks) and employs 8 quality metrics. 2. **Standardized Metrics:** WAVES prioritizes the True Positive Rate (TPR) at a stringent False Positive Rate (FPR) threshold of 0.1%, addressing the inadequacies of alternative metrics like p-values and AUROC. 3. **Performance vs. Quality Plots:** WAVES introduces Performance vs. Quality 2D plots to comprehensively compare watermark performance and image quality, providing a unified perspective on robustness. 4. **Attack Effectiveness:** WAVES ranks attacks based on their impact on detection performance and image quality, offering insights into the effectiveness of different attacks against various watermarking methods. **Evaluation Results:** - **Watermark Robustness:** StegaStamp demonstrates exceptional robustness, followed by Tree-Ring, while Stable Signature shows the least robustness. - **Attack Effectiveness:** Different watermarking methods exhibit varying vulnerabilities to different types of attacks. For example, Tree-Ring is particularly vulnerable to adversarial attacks, while Stable Signature is susceptible to most regeneration attacks. - **User Identification:** Similar trends in watermark robustness and attack effectiveness are observed in user identification tasks, with watermarks becoming more vulnerable as the number of users increases. **Limitations and Future Directions:** - **Public VAEs:** WAVES highlights the risks of using publicly available VAEs in watermarked diffusion models, as these can be easily compromised by adversarial attacks. - **Future Strategies:** Potential strategies to improve robustness include incorporating more types of transformations, improving algorithmic frameworks, and using redundant bits or hybrid approaches. **Impact Statement:** WAVES contributes to the development of more robust watermarking systems, which are crucial for protecting creative ownership and preventing the misrepresentation of AI-generated content.