MULTI TEST: Physical-Aware Object Insertion for Testing Multi-sensor Fusion Perception Systems

MULTI TEST: Physical-Aware Object Insertion for Testing Multi-sensor Fusion Perception Systems

April 14-20, 2024 | Xinyu Gao, Zhijie Wang, Yang Feng, Lei Ma, Zhenyu Chen, Baowen Xu
MultiTest is a fitness-guided metamorphic testing method for multi-sensor fusion (MSF) perception systems. It generates realistic, multi-modal test data by inserting objects into critical positions of background images and point clouds, ensuring modality consistency. A fitness metric guides the test generation process, enabling the detection of diverse faults in MSF systems. The method is implemented as the MULTI-TEST tool, which is publicly available. Experiments on five SOTA perception systems show that MULTI-TEST can generate realistic and modality-consistent test data and effectively detect hundreds of faults. Retraining MSF systems with MULTI-TEST-generated test data improves their robustness. The tool also demonstrates generalization across single-sensor systems. The results indicate that MULTI-TEST is effective in generating challenging test cases and improving system performance. The method addresses two critical challenges in testing MSF systems: modality consistency and realism of test cases. The approach leverages physical-aware object insertion and metamorphic relations to generate test data that reveals errors in perception systems. The fitness-guided testing process enhances test efficiency by selecting high-fault-revealing test cases. The experiments confirm that MULTI-TEST outperforms existing methods in generating realistic data and detecting faults. The results also show that retraining systems with MULTI-TEST-generated test data improves their performance. The method is evaluated on various perception systems and demonstrates its effectiveness in improving system robustness and fault detection.MultiTest is a fitness-guided metamorphic testing method for multi-sensor fusion (MSF) perception systems. It generates realistic, multi-modal test data by inserting objects into critical positions of background images and point clouds, ensuring modality consistency. A fitness metric guides the test generation process, enabling the detection of diverse faults in MSF systems. The method is implemented as the MULTI-TEST tool, which is publicly available. Experiments on five SOTA perception systems show that MULTI-TEST can generate realistic and modality-consistent test data and effectively detect hundreds of faults. Retraining MSF systems with MULTI-TEST-generated test data improves their robustness. The tool also demonstrates generalization across single-sensor systems. The results indicate that MULTI-TEST is effective in generating challenging test cases and improving system performance. The method addresses two critical challenges in testing MSF systems: modality consistency and realism of test cases. The approach leverages physical-aware object insertion and metamorphic relations to generate test data that reveals errors in perception systems. The fitness-guided testing process enhances test efficiency by selecting high-fault-revealing test cases. The experiments confirm that MULTI-TEST outperforms existing methods in generating realistic data and detecting faults. The results also show that retraining systems with MULTI-TEST-generated test data improves their performance. The method is evaluated on various perception systems and demonstrates its effectiveness in improving system robustness and fault detection.
Reach us at info@futurestudyspace.com