4K4DGen: Panoramic 4D Generation at 4K Resolution

4K4DGen: Panoramic 4D Generation at 4K Resolution

4 Jul 2024 | Renjie Li, Panwang Pan, Bangbang Yang, Dejia Xu, Shijie Zhou, Xuanyang Zhang, Zeming Li, Achuta Kadambi, Zhangyang Wang, Zhiwen Fan
4K4DGen is a novel framework designed to generate high-quality, immersive 4D panoramic environments at a resolution of 4096×2048. The method addresses the challenge of creating dynamic, 360-degree panoramic scenes by first animating a static panoramic image into a panoramic video using a Panoramic Denoiser, and then elevating this video into a 4D environment through a 4D lifting mechanism. The Panoramic Denoiser utilizes a pre-trained 2D diffusion model to denoise spherical latent codes, ensuring consistent animation across different views. The 4D lifting mechanism corrects scale discrepancies and ensures spatial and temporal consistency by employing a depth estimator and time-dependent 3D Gaussians. This approach enables the generation of coherent and seamless 4D panoramic rendering, even without large-scale annotated 4D data. The paper also includes experimental results and ablation studies to validate the effectiveness of the proposed methods.4K4DGen is a novel framework designed to generate high-quality, immersive 4D panoramic environments at a resolution of 4096×2048. The method addresses the challenge of creating dynamic, 360-degree panoramic scenes by first animating a static panoramic image into a panoramic video using a Panoramic Denoiser, and then elevating this video into a 4D environment through a 4D lifting mechanism. The Panoramic Denoiser utilizes a pre-trained 2D diffusion model to denoise spherical latent codes, ensuring consistent animation across different views. The 4D lifting mechanism corrects scale discrepancies and ensures spatial and temporal consistency by employing a depth estimator and time-dependent 3D Gaussians. This approach enables the generation of coherent and seamless 4D panoramic rendering, even without large-scale annotated 4D data. The paper also includes experimental results and ablation studies to validate the effectiveness of the proposed methods.
Reach us at info@study.space