4 Jul 2024 | Yiang Shi, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Xinggang Wang
This paper introduces a novel point-based representation for 3D occupancy prediction from multi-view images, named Occupancy as Set of Points (OSP). Unlike traditional volume-based methods that use uniform grids, OSP represents the scene as a set of Points of Interest (PoIs), enabling flexible sampling during training and inference. The method uses a point-based decoder to predict occupancy, allowing for adaptive sampling and enhanced adaptability. OSP outperforms existing methods in terms of performance and flexibility, and can be seamlessly integrated with volume-based methods to improve their effectiveness. Experiments on the Occ3D-nuScenes benchmark show that OSP achieves a high mIoU score of 39.4, demonstrating its strong performance and flexibility. The method is designed to handle complex scenes and adapt to different inference requirements, making it suitable for autonomous driving applications. OSP introduces a flexible framework that allows for inference at any area of interest without retraining, and can predict areas beyond the scene. The method also includes a plugin module that enhances the performance of existing volume-based methods. The key contributions of the paper include a novel point-based occupancy representation, a flexible framework for 3D occupancy prediction, and a plugin module that enhances the performance of volume-based methods. The results show that OSP achieves strong performance in 3D occupancy prediction, making it a promising approach for autonomous driving systems.This paper introduces a novel point-based representation for 3D occupancy prediction from multi-view images, named Occupancy as Set of Points (OSP). Unlike traditional volume-based methods that use uniform grids, OSP represents the scene as a set of Points of Interest (PoIs), enabling flexible sampling during training and inference. The method uses a point-based decoder to predict occupancy, allowing for adaptive sampling and enhanced adaptability. OSP outperforms existing methods in terms of performance and flexibility, and can be seamlessly integrated with volume-based methods to improve their effectiveness. Experiments on the Occ3D-nuScenes benchmark show that OSP achieves a high mIoU score of 39.4, demonstrating its strong performance and flexibility. The method is designed to handle complex scenes and adapt to different inference requirements, making it suitable for autonomous driving applications. OSP introduces a flexible framework that allows for inference at any area of interest without retraining, and can predict areas beyond the scene. The method also includes a plugin module that enhances the performance of existing volume-based methods. The key contributions of the paper include a novel point-based occupancy representation, a flexible framework for 3D occupancy prediction, and a plugin module that enhances the performance of volume-based methods. The results show that OSP achieves strong performance in 3D occupancy prediction, making it a promising approach for autonomous driving systems.