The paper "On Evaluating Adversarial Robustness" addresses the challenges of evaluating defenses against adversarial examples, which have proven difficult due to the complexity of security evaluations. The authors, from Google Brain, MIT, and the University of Tübingen, provide methodological foundations, review best practices, and suggest new methods for evaluating defenses. They emphasize the importance of a clear threat model, which specifies the conditions under which a defense is designed to be secure, and the need for skepticism in evaluating results. The paper also includes a checklist to avoid common evaluation pitfalls, such as not mindlessly following the checklist, applying a diverse set of attacks, and focusing on the strongest attacks possible. Additionally, it recommends investigating provable approaches, reporting clean model accuracy, and performing a transferability analysis to validate the defense's effectiveness. The authors stress the importance of reproducible research by releasing full source code and pre-trained models to ensure the correctness of evaluations.The paper "On Evaluating Adversarial Robustness" addresses the challenges of evaluating defenses against adversarial examples, which have proven difficult due to the complexity of security evaluations. The authors, from Google Brain, MIT, and the University of Tübingen, provide methodological foundations, review best practices, and suggest new methods for evaluating defenses. They emphasize the importance of a clear threat model, which specifies the conditions under which a defense is designed to be secure, and the need for skepticism in evaluating results. The paper also includes a checklist to avoid common evaluation pitfalls, such as not mindlessly following the checklist, applying a diverse set of attacks, and focusing on the strongest attacks possible. Additionally, it recommends investigating provable approaches, reporting clean model accuracy, and performing a transferability analysis to validate the defense's effectiveness. The authors stress the importance of reproducible research by releasing full source code and pre-trained models to ensure the correctness of evaluations.