28 Mar 2024 | Lingjun Zhao*, Jingyu Song†, Katherine A. Skinner
CRKD is a novel cross-modality knowledge distillation framework for 3D object detection that bridges the performance gap between LiDAR-Camera (LC) and Camera-Radar (CR) detectors. The framework uses a shared Bird's-Eye-View (BEV) feature space to enable effective knowledge distillation from an LC teacher model to a CR student model. Four distillation losses are designed to address the discrepancies between different sensors and facilitate effective cross-modality knowledge distillation. The framework is evaluated on the nuScenes dataset, demonstrating significant improvements in mAP and NDS for CR detectors. CRKD is the first KD framework that supports a fusion-to-fusion distillation path, leveraging the shared BEV feature space and point cloud representation between LiDAR and radar measurements. The framework improves the detection performance of CR detectors by enabling adaptive fusion and leveraging the strengths of radar for dynamic object detection. The proposed CRKD framework is effective in improving the performance of CR detectors and has the potential to facilitate the practical application of perceptual autonomy with a low-cost and robust CR sensor configuration.CRKD is a novel cross-modality knowledge distillation framework for 3D object detection that bridges the performance gap between LiDAR-Camera (LC) and Camera-Radar (CR) detectors. The framework uses a shared Bird's-Eye-View (BEV) feature space to enable effective knowledge distillation from an LC teacher model to a CR student model. Four distillation losses are designed to address the discrepancies between different sensors and facilitate effective cross-modality knowledge distillation. The framework is evaluated on the nuScenes dataset, demonstrating significant improvements in mAP and NDS for CR detectors. CRKD is the first KD framework that supports a fusion-to-fusion distillation path, leveraging the shared BEV feature space and point cloud representation between LiDAR and radar measurements. The framework improves the detection performance of CR detectors by enabling adaptive fusion and leveraging the strengths of radar for dynamic object detection. The proposed CRKD framework is effective in improving the performance of CR detectors and has the potential to facilitate the practical application of perceptual autonomy with a low-cost and robust CR sensor configuration.