Feature Splatting: Language-Driven Physics-Based Scene Synthesis and Editing

Feature Splatting: Language-Driven Physics-Based Scene Synthesis and Editing

1 Apr 2024 | Ri-Zhao Qiu, Ge Yang, Weijia Zeng, Xiaolong Wang
**Feature Splatting** is an innovative approach that integrates physics-based dynamic scene synthesis with rich semantics from vision-language models, grounded by natural language queries. The method addresses the challenge of manipulating both the appearance and physical properties of objects in 3D scenes. Key contributions include: 1. **Distilling Vision-Language Features into 3D Gaussians**: This enables semi-automatic scene decomposition using text queries, allowing for the automatic assignment of material properties. 2. **Physics-Based Dynamic Scene Synthesis**: The method uses a particle-based simulator to synthesize dynamic interactions, where material properties are assigned via text queries. 3. **Unified Representation**: Feature Splatting unifies photo-realism, rich semantics, and physics-based dynamic synthesis in a single format. The pipeline includes: - **Differentiable Feature Splatting**: Representing scenes as a mixture of 3D Gaussians, with additional vectors for semantics. - **Language-Guided Scene Decomposition**: Using open-text queries to segment objects and determine their physical properties. - **Language-Driven Physics Synthesis**: Assigning physical properties and estimating collision surfaces and gravity for realistic dynamics. The paper also discusses technical challenges and solutions, such as improving feature quality using part-priors and regularizing feature maps. Experiments demonstrate the effectiveness of Feature Splatting in editing and synthesizing dynamic scenes, with qualitative and quantitative results showing its ability to handle complex tasks like volume preservation and temporal continuity.**Feature Splatting** is an innovative approach that integrates physics-based dynamic scene synthesis with rich semantics from vision-language models, grounded by natural language queries. The method addresses the challenge of manipulating both the appearance and physical properties of objects in 3D scenes. Key contributions include: 1. **Distilling Vision-Language Features into 3D Gaussians**: This enables semi-automatic scene decomposition using text queries, allowing for the automatic assignment of material properties. 2. **Physics-Based Dynamic Scene Synthesis**: The method uses a particle-based simulator to synthesize dynamic interactions, where material properties are assigned via text queries. 3. **Unified Representation**: Feature Splatting unifies photo-realism, rich semantics, and physics-based dynamic synthesis in a single format. The pipeline includes: - **Differentiable Feature Splatting**: Representing scenes as a mixture of 3D Gaussians, with additional vectors for semantics. - **Language-Guided Scene Decomposition**: Using open-text queries to segment objects and determine their physical properties. - **Language-Driven Physics Synthesis**: Assigning physical properties and estimating collision surfaces and gravity for realistic dynamics. The paper also discusses technical challenges and solutions, such as improving feature quality using part-priors and regularizing feature maps. Experiments demonstrate the effectiveness of Feature Splatting in editing and synthesizing dynamic scenes, with qualitative and quantitative results showing its ability to handle complex tasks like volume preservation and temporal continuity.
Reach us at info@study.space
[slides] Feature Splatting%3A Language-Driven Physics-Based Scene Synthesis and Editing | StudySpace