Contrastive Gaussian Clustering: Weakly Supervised 3D Scene Segmentation

Contrastive Gaussian Clustering: Weakly Supervised 3D Scene Segmentation

19 Apr 2024 | Myrna C. Silva*, Mahtab Dahaghin*, Matteo Toso, and Alessio Del Bue
The paper introduces *Contrastive Gaussian Clustering* (CGC), a novel approach for 3D scene segmentation that leverages 3D Gaussian Splatting (3DGS) and contrastive learning to generate accurate segmentation masks from any viewpoint. The method trains a model to include segmentation feature vectors for each 3D Gaussian, enabling 3D scene segmentation by clustering Gaussians according to their feature vectors. The model can also generate 2D segmentation masks by projecting Gaussians onto a plane and α-blending their segmentation features. By combining contrastive learning and spatial regularization, CGC can handle inconsistent 2D segmentation masks and learn consistent 3D feature fields across all views. The method outperforms state-of-the-art approaches in terms of intersection over union (IoU) accuracy, improving by +8%. The paper includes experiments on two datasets, LERF-Mask and 3D-OVS, demonstrating the effectiveness of CGC in generating accurate and consistent segmentation masks.The paper introduces *Contrastive Gaussian Clustering* (CGC), a novel approach for 3D scene segmentation that leverages 3D Gaussian Splatting (3DGS) and contrastive learning to generate accurate segmentation masks from any viewpoint. The method trains a model to include segmentation feature vectors for each 3D Gaussian, enabling 3D scene segmentation by clustering Gaussians according to their feature vectors. The model can also generate 2D segmentation masks by projecting Gaussians onto a plane and α-blending their segmentation features. By combining contrastive learning and spatial regularization, CGC can handle inconsistent 2D segmentation masks and learn consistent 3D feature fields across all views. The method outperforms state-of-the-art approaches in terms of intersection over union (IoU) accuracy, improving by +8%. The paper includes experiments on two datasets, LERF-Mask and 3D-OVS, demonstrating the effectiveness of CGC in generating accurate and consistent segmentation masks.
Reach us at info@study.space