Collaborative Semantic Occupancy Prediction with Hybrid Feature Fusion in Connected Automated Vehicles

Collaborative Semantic Occupancy Prediction with Hybrid Feature Fusion in Connected Automated Vehicles

25 Apr 2024 | Rui Song, Chenwei Liang, Hu Cao, Zhiran Yan, Walter Zimmer, Markus Gross, Andreas Festag, Alois Knoll
The paper introduces the Collaborative Hybrid Feature Fusion (CoHFF) framework for collaborative semantic occupancy prediction in connected automated vehicles (CAVs). The framework leverages the exchange of information between CAVs to enhance perception accuracy and completeness. Key contributions include: 1. **First Camera-Based Framework**: CoHFF is the first camera-based framework for collaborative semantic occupancy prediction, improving local 3D semantic occupancy predictions by hybrid fusion of semantic and occupancy task features, and compressed orthogonal attention features shared among vehicles. 2. **Hybrid Feature Fusion**: The framework combines features from both semantic and occupancy tasks, enhancing performance over models trained solely for one task. This fusion process involves inter-CAV semantic information fusion via V2X Feature Fusion and intra-CAV fusion of semantic information with occupancy status through task feature fusion. 3. **Dataset Augmentation**: To evaluate the framework, the OPV2V dataset is extended with 3D collaborative semantic occupancy labels, providing a more robust evaluation environment. 4. **Experimental Results**: Experiments show that CoHFF outperforms single-vehicle performance by over 30% in most categories and outperforms state-of-the-art collaborative 3D detection techniques in downstream applications, demonstrating enhanced accuracy and enriched semantic awareness in road environments. 5. **Ablation Study**: Ablation studies validate the effectiveness of each component of the CoHFF framework, showing that collaboration significantly improves semantic occupancy prediction. 6. **Robustness**: The framework demonstrates robust performance under low communication budgets and varying GPS noise levels, further validating its practical applicability. Overall, CoHFF provides a comprehensive solution for collaborative semantic occupancy prediction, enhancing the perception capabilities of CAVs in complex road environments.The paper introduces the Collaborative Hybrid Feature Fusion (CoHFF) framework for collaborative semantic occupancy prediction in connected automated vehicles (CAVs). The framework leverages the exchange of information between CAVs to enhance perception accuracy and completeness. Key contributions include: 1. **First Camera-Based Framework**: CoHFF is the first camera-based framework for collaborative semantic occupancy prediction, improving local 3D semantic occupancy predictions by hybrid fusion of semantic and occupancy task features, and compressed orthogonal attention features shared among vehicles. 2. **Hybrid Feature Fusion**: The framework combines features from both semantic and occupancy tasks, enhancing performance over models trained solely for one task. This fusion process involves inter-CAV semantic information fusion via V2X Feature Fusion and intra-CAV fusion of semantic information with occupancy status through task feature fusion. 3. **Dataset Augmentation**: To evaluate the framework, the OPV2V dataset is extended with 3D collaborative semantic occupancy labels, providing a more robust evaluation environment. 4. **Experimental Results**: Experiments show that CoHFF outperforms single-vehicle performance by over 30% in most categories and outperforms state-of-the-art collaborative 3D detection techniques in downstream applications, demonstrating enhanced accuracy and enriched semantic awareness in road environments. 5. **Ablation Study**: Ablation studies validate the effectiveness of each component of the CoHFF framework, showing that collaboration significantly improves semantic occupancy prediction. 6. **Robustness**: The framework demonstrates robust performance under low communication budgets and varying GPS noise levels, further validating its practical applicability. Overall, CoHFF provides a comprehensive solution for collaborative semantic occupancy prediction, enhancing the perception capabilities of CAVs in complex road environments.
Reach us at info@study.space