Feature Splatting: Language-Driven Physics-Based Scene Synthesis and Editing

Feature Splatting: Language-Driven Physics-Based Scene Synthesis and Editing

1 Apr 2024 | Ri-Zhao Qiu¹, Ge Yang²,³, Weijia Zeng¹, and Xiaolong Wang¹
Feature Splatting is a method that combines physics-based dynamic scene synthesis with rich semantics from vision-language foundation models grounded in natural language. The approach allows for semi-automated scene decomposition using text queries and synthesizes physics-based dynamics from static scenes using a particle-based simulator. Key contributions include distilling high-quality, object-centric vision-language features into 3D Gaussians, enabling language-driven scene editing, and using a physics engine adapted to Gaussian-based representations for dynamic synthesis. The method addresses challenges in using feature-carrying 3D Gaussians as a unified format for appearance, geometry, material properties, and semantics. Feature Splatting supports language-guided scene decomposition, where objects are segmented and material properties are assigned via text queries. It also enables physics-based dynamic synthesis by integrating material-point methods with Gaussian representations. The method improves feature quality by using part-level masks and regularization techniques, and it supports real-time rendering and efficient training. Experiments demonstrate the effectiveness of Feature Splatting in dynamic scene synthesis, appearance editing, and geometry manipulation. The method outperforms existing approaches in terms of localization accuracy and rendering quality, and it enables semi-automated scene editing with language queries. The system is optimized for efficiency, with training times reduced by over 60% compared to baselines. Feature Splatting provides a unified representation for appearance, geometry, material properties, and semantics, enabling language-driven physics-based scene synthesis and editing.Feature Splatting is a method that combines physics-based dynamic scene synthesis with rich semantics from vision-language foundation models grounded in natural language. The approach allows for semi-automated scene decomposition using text queries and synthesizes physics-based dynamics from static scenes using a particle-based simulator. Key contributions include distilling high-quality, object-centric vision-language features into 3D Gaussians, enabling language-driven scene editing, and using a physics engine adapted to Gaussian-based representations for dynamic synthesis. The method addresses challenges in using feature-carrying 3D Gaussians as a unified format for appearance, geometry, material properties, and semantics. Feature Splatting supports language-guided scene decomposition, where objects are segmented and material properties are assigned via text queries. It also enables physics-based dynamic synthesis by integrating material-point methods with Gaussian representations. The method improves feature quality by using part-level masks and regularization techniques, and it supports real-time rendering and efficient training. Experiments demonstrate the effectiveness of Feature Splatting in dynamic scene synthesis, appearance editing, and geometry manipulation. The method outperforms existing approaches in terms of localization accuracy and rendering quality, and it enables semi-automated scene editing with language queries. The system is optimized for efficiency, with training times reduced by over 60% compared to baselines. Feature Splatting provides a unified representation for appearance, geometry, material properties, and semantics, enabling language-driven physics-based scene synthesis and editing.
Reach us at info@study.space