The paper introduces AutoAttack, an ensemble of parameter-free attacks for evaluating adversarial robustness. It addresses the issue of unreliable evaluation of adversarial defenses, which often overestimate robustness. The authors propose two improvements to the PGD attack: Auto-PGD, which adapts step sizes automatically, and a new loss function, the Difference of Logits Ratio (DLR), which is shift and rescaling invariant. These are combined with two existing attacks, FAB and Square Attack, to form AutoAttack, a parameter-free, computationally efficient, and user-independent ensemble. AutoAttack is tested on over 50 models from recent top conferences, achieving lower robust test accuracy than reported in most cases, often by more than 10%, identifying several broken defenses. The ensemble's diversity helps in evaluating robustness reliably, as different attacks may fail on some models but succeed on others. AutoAttack is shown to be effective for both deterministic and randomized defenses, and it is recommended as a standard evaluation method due to its reliability and low computational cost. The paper highlights the importance of using diverse and parameter-free attacks to accurately assess the robustness of adversarial defenses.The paper introduces AutoAttack, an ensemble of parameter-free attacks for evaluating adversarial robustness. It addresses the issue of unreliable evaluation of adversarial defenses, which often overestimate robustness. The authors propose two improvements to the PGD attack: Auto-PGD, which adapts step sizes automatically, and a new loss function, the Difference of Logits Ratio (DLR), which is shift and rescaling invariant. These are combined with two existing attacks, FAB and Square Attack, to form AutoAttack, a parameter-free, computationally efficient, and user-independent ensemble. AutoAttack is tested on over 50 models from recent top conferences, achieving lower robust test accuracy than reported in most cases, often by more than 10%, identifying several broken defenses. The ensemble's diversity helps in evaluating robustness reliably, as different attacks may fail on some models but succeed on others. AutoAttack is shown to be effective for both deterministic and randomized defenses, and it is recommended as a standard evaluation method due to its reliability and low computational cost. The paper highlights the importance of using diverse and parameter-free attacks to accurately assess the robustness of adversarial defenses.