27 May 2024 | Anran Liu, Cheng Lin, Yuan Liu, Xiaoxiao Long, Zhiyang Dou, Hao-Xiang Guo, Ping Luo, Wenping Wang
The paper introduces Part123, a novel framework for part-aware 3D reconstruction from a single-view image. The method first generates multiview-consistent images using diffusion models and then leverages the Segment Anything Model (SAM) to predict 2D segmentation masks. To handle the inconsistency and lack of correspondence between masks across views, contrastive learning is introduced into a neural rendering framework to learn a part-aware feature space. An automatic algorithm is also developed to determine the number of parts based on multiview SAM predictions. Experiments demonstrate that Part123 can generate high-quality 3D models with segmented parts on various objects, outperforming existing methods in terms of reconstruction quality and part segmentation accuracy. The part-aware 3D models generated by Part123 are beneficial for applications such as feature-preserving reconstruction, primitive fitting, and 3D shape editing.The paper introduces Part123, a novel framework for part-aware 3D reconstruction from a single-view image. The method first generates multiview-consistent images using diffusion models and then leverages the Segment Anything Model (SAM) to predict 2D segmentation masks. To handle the inconsistency and lack of correspondence between masks across views, contrastive learning is introduced into a neural rendering framework to learn a part-aware feature space. An automatic algorithm is also developed to determine the number of parts based on multiview SAM predictions. Experiments demonstrate that Part123 can generate high-quality 3D models with segmented parts on various objects, outperforming existing methods in terms of reconstruction quality and part segmentation accuracy. The part-aware 3D models generated by Part123 are beneficial for applications such as feature-preserving reconstruction, primitive fitting, and 3D shape editing.