Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Video

Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Video

15 Apr 2024 | Hongchi Xia, Zhi-Hao Lin, Wei-Chiu Ma, Shenlong Wang
Video2Game is a novel approach that automatically converts real-world video footage into interactive, realistic, and browser-compatible game environments. The system consists of three core components: a neural radiance fields (NeRF) module for capturing scene geometry and appearance, a mesh module for efficient rendering, and a physics module for modeling interactions. By combining these components, Video2Game can construct a digital replica of a scene that supports free exploration and interaction with objects. The system is benchmarked on both indoor and large-scale outdoor scenes, demonstrating high-quality real-time rendering and interactive capabilities. Key contributions include a novel 3D modeling algorithm for real-time rendering and physical simulation, and an automated game-creation framework that integrates with common game engines. The paper also discusses related work in novel view synthesis, controllable video generation, and data-driven simulation, highlighting the unique aspects of Video2Game's approach. Experimental results show superior performance in rendering quality and interactive compatibility compared to state-of-the-art methods.Video2Game is a novel approach that automatically converts real-world video footage into interactive, realistic, and browser-compatible game environments. The system consists of three core components: a neural radiance fields (NeRF) module for capturing scene geometry and appearance, a mesh module for efficient rendering, and a physics module for modeling interactions. By combining these components, Video2Game can construct a digital replica of a scene that supports free exploration and interaction with objects. The system is benchmarked on both indoor and large-scale outdoor scenes, demonstrating high-quality real-time rendering and interactive capabilities. Key contributions include a novel 3D modeling algorithm for real-time rendering and physical simulation, and an automated game-creation framework that integrates with common game engines. The paper also discusses related work in novel view synthesis, controllable video generation, and data-driven simulation, highlighting the unique aspects of Video2Game's approach. Experimental results show superior performance in rendering quality and interactive compatibility compared to state-of-the-art methods.
Reach us at info@study.space
[slides] Video2Game%3A Real-time%2C Interactive%2C Realistic and Browser-Compatible Environment from a Single Video | StudySpace