**DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars** **Authors:** Yuchi Tian, Suman Jana, Kexin Pei, Baishakhi Ray **Abstract:** Recent advancements in Deep Neural Networks (DNNs) have led to the development of autonomous vehicles that can drive without human intervention. However, DNNs often exhibit incorrect or unexpected behaviors under certain conditions, which can lead to fatal accidents. Existing testing techniques for DNN-driven vehicles are costly and inefficient. This paper introduces DeepTest, a systematic testing tool designed to automatically detect erroneous behaviors in DNN-driven vehicles. DeepTest generates test cases by maximizing neuron coverage, which measures the number of neurons activated by a set of inputs. The tool explores different driving conditions, such as rain, fog, and lighting changes, to find thousands of erroneous behaviors in three top-performing DNNs from the Udacity self-driving car challenge. DeepTest also demonstrates that synthetic images generated by applying realistic image transformations can increase neuron coverage by up to 100%. The tool uses metamorphic relations to automatically detect erroneous behaviors and has been evaluated on three DNN models, identifying over 6,000 erroneous behaviors with low false positives. **Contributions:** 1. **Systematic Testing with Neuron Coverage:** DeepTest uses neuron coverage to partition the input space and generate test cases that maximize coverage. 2. **Increasing Coverage with Synthetic Images:** DeepTest generates realistic synthetic images by applying various image transformations to seed images, increasing neuron coverage. 3. **Creating a Test Oracle with Metamorphic Relations:** DeepTest leverages metamorphic relations to automatically detect erroneous behaviors without detailed manual specifications. **Methods:** - **Neuron Coverage:** Measures the ratio of activated neurons to total neurons. - **Image Transformations:** Includes linear, affine, and convolutional transformations to mimic real-world conditions. - **Combining Transformations:** Uses a greedy search technique to combine transformations to increase neuron coverage. - **Metamorphic Relations:** Defines relationships between car behaviors across different synthetic images to detect erroneous behaviors. **Results:** - **Neuron Coverage Correlation:** neuron coverage is correlated with input-output diversity. - **Image Transformations:** Different transformations activate different sets of neurons. - **Cumulative Transformations:** Combined transformations increase neuron coverage. - **Metamorphic Relations:** DeepTest successfully detects erroneous behaviors using metamorphic relations. - **Retraining DNNs:** Retraining DNNs with synthetic images improves accuracy by up to 46%. **Related Work:** - **Testing of Driver Assistance Systems:** Focuses on warning logic and critical scenarios. - **Testing and Verification of Machine Learning:** Uses blackbox testing and ad hoc simulations. - **Adversarial Machine Learning:** Attacks DNNs by forcing incorrect predictions. - **Test Amplification:** Generates and amplifies test cases for traditional software. - **Metamorphic Testing:****DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars** **Authors:** Yuchi Tian, Suman Jana, Kexin Pei, Baishakhi Ray **Abstract:** Recent advancements in Deep Neural Networks (DNNs) have led to the development of autonomous vehicles that can drive without human intervention. However, DNNs often exhibit incorrect or unexpected behaviors under certain conditions, which can lead to fatal accidents. Existing testing techniques for DNN-driven vehicles are costly and inefficient. This paper introduces DeepTest, a systematic testing tool designed to automatically detect erroneous behaviors in DNN-driven vehicles. DeepTest generates test cases by maximizing neuron coverage, which measures the number of neurons activated by a set of inputs. The tool explores different driving conditions, such as rain, fog, and lighting changes, to find thousands of erroneous behaviors in three top-performing DNNs from the Udacity self-driving car challenge. DeepTest also demonstrates that synthetic images generated by applying realistic image transformations can increase neuron coverage by up to 100%. The tool uses metamorphic relations to automatically detect erroneous behaviors and has been evaluated on three DNN models, identifying over 6,000 erroneous behaviors with low false positives. **Contributions:** 1. **Systematic Testing with Neuron Coverage:** DeepTest uses neuron coverage to partition the input space and generate test cases that maximize coverage. 2. **Increasing Coverage with Synthetic Images:** DeepTest generates realistic synthetic images by applying various image transformations to seed images, increasing neuron coverage. 3. **Creating a Test Oracle with Metamorphic Relations:** DeepTest leverages metamorphic relations to automatically detect erroneous behaviors without detailed manual specifications. **Methods:** - **Neuron Coverage:** Measures the ratio of activated neurons to total neurons. - **Image Transformations:** Includes linear, affine, and convolutional transformations to mimic real-world conditions. - **Combining Transformations:** Uses a greedy search technique to combine transformations to increase neuron coverage. - **Metamorphic Relations:** Defines relationships between car behaviors across different synthetic images to detect erroneous behaviors. **Results:** - **Neuron Coverage Correlation:** neuron coverage is correlated with input-output diversity. - **Image Transformations:** Different transformations activate different sets of neurons. - **Cumulative Transformations:** Combined transformations increase neuron coverage. - **Metamorphic Relations:** DeepTest successfully detects erroneous behaviors using metamorphic relations. - **Retraining DNNs:** Retraining DNNs with synthetic images improves accuracy by up to 46%. **Related Work:** - **Testing of Driver Assistance Systems:** Focuses on warning logic and critical scenarios. - **Testing and Verification of Machine Learning:** Uses blackbox testing and ad hoc simulations. - **Adversarial Machine Learning:** Attacks DNNs by forcing incorrect predictions. - **Test Amplification:** Generates and amplifies test cases for traditional software. - **Metamorphic Testing:**