The paper "Revisiting Adversarial Training at Scale" by Zeyu Wang, Xianhang Li, Hongru Zhu, and Cihang Xie from UC Santa Cruz addresses the challenge of adversarial training in the context of large-scale models and datasets. The authors introduce a novel framework called AdvXL, which enables efficient and effective adversarial training for models with billions of parameters and web-scale datasets. **Key Contributions:** 1. **AdvXL Framework:** AdvXL is designed to scale adversarial training to large models and datasets while maintaining computational affordability. It employs a two-stage training process: lightweight pre-training followed by intensive fine-tuning. 2. **Efficiency and Effectiveness:** The framework significantly reduces computational costs compared to traditional adversarial training methods, making it feasible to train models on datasets with billions of samples. 3. **State-of-the-Art Performance:** On ImageNet-1K, AdvXL achieves state-of-the-art robust accuracy records under AutoAttack, surpassing previous methods by margins of up to 14.2% in $l_{2}$- and $l_{1}$-robust accuracy. **Methodology:** - **Two-Stage Training:** The first stage involves pre-training with reduced image token lengths and weaker attacks, while the second stage uses full-resolution inputs and stronger attacks. - **CLIP Text Encoder:** The framework leverages the CLIP text encoder to enable training on web-scale datasets with natural language captions, enhancing the model's ability to learn complex class relationships. **Experiments:** - **Model and Data Scaling:** The authors scale the model size to 1B parameters and the dataset size to over 1B samples, demonstrating significant improvements in robust accuracy. - **Ablation Studies:** Various design choices, such as image token reduction and attack strength, are evaluated to optimize the training process. - **Comparison with State-of-the-Art:** AdvXL outperforms existing methods in terms of $l_{\infty}$, $l_{2}$, and $l_{1}$-robust accuracy, showcasing its effectiveness in adversarial robustness. **Conclusion:** AdvXL represents a significant advancement in adversarial training, enabling the training of robust visual models at unprecedented scales. The framework's efficiency and effectiveness open new avenues for further research and practical applications in the field of machine learning.The paper "Revisiting Adversarial Training at Scale" by Zeyu Wang, Xianhang Li, Hongru Zhu, and Cihang Xie from UC Santa Cruz addresses the challenge of adversarial training in the context of large-scale models and datasets. The authors introduce a novel framework called AdvXL, which enables efficient and effective adversarial training for models with billions of parameters and web-scale datasets. **Key Contributions:** 1. **AdvXL Framework:** AdvXL is designed to scale adversarial training to large models and datasets while maintaining computational affordability. It employs a two-stage training process: lightweight pre-training followed by intensive fine-tuning. 2. **Efficiency and Effectiveness:** The framework significantly reduces computational costs compared to traditional adversarial training methods, making it feasible to train models on datasets with billions of samples. 3. **State-of-the-Art Performance:** On ImageNet-1K, AdvXL achieves state-of-the-art robust accuracy records under AutoAttack, surpassing previous methods by margins of up to 14.2% in $l_{2}$- and $l_{1}$-robust accuracy. **Methodology:** - **Two-Stage Training:** The first stage involves pre-training with reduced image token lengths and weaker attacks, while the second stage uses full-resolution inputs and stronger attacks. - **CLIP Text Encoder:** The framework leverages the CLIP text encoder to enable training on web-scale datasets with natural language captions, enhancing the model's ability to learn complex class relationships. **Experiments:** - **Model and Data Scaling:** The authors scale the model size to 1B parameters and the dataset size to over 1B samples, demonstrating significant improvements in robust accuracy. - **Ablation Studies:** Various design choices, such as image token reduction and attack strength, are evaluated to optimize the training process. - **Comparison with State-of-the-Art:** AdvXL outperforms existing methods in terms of $l_{\infty}$, $l_{2}$, and $l_{1}$-robust accuracy, showcasing its effectiveness in adversarial robustness. **Conclusion:** AdvXL represents a significant advancement in adversarial training, enabling the training of robust visual models at unprecedented scales. The framework's efficiency and effectiveness open new avenues for further research and practical applications in the field of machine learning.