Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning Battista Biggio, Fabio Roli Abstract: Learning-based pattern classifiers, including deep networks, have shown impressive performance in various application domains, from computer vision to cybersecurity. However, adversarial input perturbations can easily subvert their predictions. The vulnerability of machine learning to such wild patterns (adversarial examples) and the design of suitable countermeasures have been investigated in adversarial machine learning. This work provides a thorough overview of the evolution of this research area over the last ten years, starting from earlier work on the security of non-deep learning algorithms up to recent work on the security properties of deep learning algorithms in computer vision and cybersecurity tasks. It highlights common misconceptions related to the security evaluation of machine-learning algorithms and discusses the main limitations of current work and future challenges towards the design of more secure learning algorithms. Keywords: Adversarial Machine Learning; Evasion Attacks; Poisoning Attacks; Adversarial Examples; Secure Learning; Deep Learning The primary misconception is about the start date of adversarial machine learning, which is not 2014. The first seminal work in adversarial machine learning dates back to 2004. This research area has been independently developing and re-discovering well-known phenomena that had been largely explored in the field of adversarial machine learning before the discovery of adversarial examples against deep networks. The fact that adversarial machine learning was well-established before 2014 is also witnessed by related events, including the 2007 NIPS Workshop on Machine Learning in Adversarial Environments for Computer Security, the 2013 Dagstuhl Perspectives Workshop on Machine Learning Methods for Computer Security, and the 2017 AISec Workshop. A book has also been recently published on this subject. This work aims to provide a thorough overview of the evolution of this interdisciplinary research area over the last ten years and beyond, from pioneering work on the security of (non-deep) learning algorithms to more recent work focused on the security properties of deep learning algorithms in computer vision and cybersecurity tasks. Our goal is to connect the dots between these apparently-different lines of work, while also highlighting common misconceptions related to the security evaluation of learning algorithms. We first review the notion of arms race in computer security, advocating for a proactive security-by-design cycle that explicitly accounts for the presence of the attacker in the loop (Sect. 2). Our narrative of the security of machine learning then follows three metaphors, referred to as the three golden rules in the following: (i) know your adversary, (ii) be proactive; and (iii) protect yourself. Knowing the attacker amounts to modeling threats against the learning-based system under design. To this end, we review a comprehensive threat model which allows one to envision and simulate attacks against the system under design, to thoroughly assess its security properties underWild Patterns: Ten Years After the Rise of Adversarial Machine Learning Battista Biggio, Fabio Roli Abstract: Learning-based pattern classifiers, including deep networks, have shown impressive performance in various application domains, from computer vision to cybersecurity. However, adversarial input perturbations can easily subvert their predictions. The vulnerability of machine learning to such wild patterns (adversarial examples) and the design of suitable countermeasures have been investigated in adversarial machine learning. This work provides a thorough overview of the evolution of this research area over the last ten years, starting from earlier work on the security of non-deep learning algorithms up to recent work on the security properties of deep learning algorithms in computer vision and cybersecurity tasks. It highlights common misconceptions related to the security evaluation of machine-learning algorithms and discusses the main limitations of current work and future challenges towards the design of more secure learning algorithms. Keywords: Adversarial Machine Learning; Evasion Attacks; Poisoning Attacks; Adversarial Examples; Secure Learning; Deep Learning The primary misconception is about the start date of adversarial machine learning, which is not 2014. The first seminal work in adversarial machine learning dates back to 2004. This research area has been independently developing and re-discovering well-known phenomena that had been largely explored in the field of adversarial machine learning before the discovery of adversarial examples against deep networks. The fact that adversarial machine learning was well-established before 2014 is also witnessed by related events, including the 2007 NIPS Workshop on Machine Learning in Adversarial Environments for Computer Security, the 2013 Dagstuhl Perspectives Workshop on Machine Learning Methods for Computer Security, and the 2017 AISec Workshop. A book has also been recently published on this subject. This work aims to provide a thorough overview of the evolution of this interdisciplinary research area over the last ten years and beyond, from pioneering work on the security of (non-deep) learning algorithms to more recent work focused on the security properties of deep learning algorithms in computer vision and cybersecurity tasks. Our goal is to connect the dots between these apparently-different lines of work, while also highlighting common misconceptions related to the security evaluation of learning algorithms. We first review the notion of arms race in computer security, advocating for a proactive security-by-design cycle that explicitly accounts for the presence of the attacker in the loop (Sect. 2). Our narrative of the security of machine learning then follows three metaphors, referred to as the three golden rules in the following: (i) know your adversary, (ii) be proactive; and (iii) protect yourself. Knowing the attacker amounts to modeling threats against the learning-based system under design. To this end, we review a comprehensive threat model which allows one to envision and simulate attacks against the system under design, to thoroughly assess its security properties under