DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars

DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars

May 27-June 3, 2018 | Yuchi Tian, Kexin Pei, Suman Jana, Baishakhi Ray
DeepTest is an automated testing tool designed to detect erroneous behaviors in deep neural network (DNN)-driven autonomous vehicles. As DNNs become increasingly prevalent in autonomous cars, they face challenges in ensuring safety due to their complex and unpredictable behaviors under various driving conditions. Traditional testing methods are often manual and expensive, making it difficult to cover all possible scenarios. DeepTest addresses this by systematically generating test cases that maximize neuron coverage, a metric that reflects the activation of neurons in the DNN. This approach helps identify corner-case behaviors that could lead to fatal collisions. DeepTest leverages real-world driving conditions such as rain, fog, and lighting changes to generate synthetic test images. These images are transformed using various techniques to activate different sets of neurons, thereby increasing coverage. By combining these transformations, DeepTest can significantly enhance neuron coverage compared to manual test inputs. Additionally, DeepTest uses metamorphic relations to automatically detect erroneous behaviors by comparing the outputs of transformed images with their original counterparts. The tool was evaluated on three top-performing DNN models from the Udacity self-driving challenge, revealing thousands of erroneous behaviors. These findings highlight the importance of automated testing in ensuring the safety and reliability of DNN-driven autonomous vehicles. DeepTest's approach not only helps in identifying potential issues but also contributes to improving the robustness of DNNs through retraining with synthetic data. The results demonstrate that DeepTest can effectively detect a large number of erroneous behaviors with low false positives, making it a valuable tool for the development and testing of autonomous vehicles.DeepTest is an automated testing tool designed to detect erroneous behaviors in deep neural network (DNN)-driven autonomous vehicles. As DNNs become increasingly prevalent in autonomous cars, they face challenges in ensuring safety due to their complex and unpredictable behaviors under various driving conditions. Traditional testing methods are often manual and expensive, making it difficult to cover all possible scenarios. DeepTest addresses this by systematically generating test cases that maximize neuron coverage, a metric that reflects the activation of neurons in the DNN. This approach helps identify corner-case behaviors that could lead to fatal collisions. DeepTest leverages real-world driving conditions such as rain, fog, and lighting changes to generate synthetic test images. These images are transformed using various techniques to activate different sets of neurons, thereby increasing coverage. By combining these transformations, DeepTest can significantly enhance neuron coverage compared to manual test inputs. Additionally, DeepTest uses metamorphic relations to automatically detect erroneous behaviors by comparing the outputs of transformed images with their original counterparts. The tool was evaluated on three top-performing DNN models from the Udacity self-driving challenge, revealing thousands of erroneous behaviors. These findings highlight the importance of automated testing in ensuring the safety and reliability of DNN-driven autonomous vehicles. DeepTest's approach not only helps in identifying potential issues but also contributes to improving the robustness of DNNs through retraining with synthetic data. The results demonstrate that DeepTest can effectively detect a large number of erroneous behaviors with low false positives, making it a valuable tool for the development and testing of autonomous vehicles.
Reach us at info@study.space