10 Oct 2024 | NICOLAS MOENNE-LOCCOZ†, NVIDIA, Canada ASHKAN MIRZAEI†, NVIDIA, Canada and University of Toronto, Canada OR PEREL, NVIDIA, Israel RICCARDO DE LUTIO, NVIDIA, USA JANICK MARTINEZ ESTURO, NVIDIA, Germany GAVRIEL STATE, NVIDIA, Canada SANJA FIDLER, NVIDIA, Canada, University of Toronto, Canada, and Vector Institute, Canada NICHOLAS SHARP†, NVIDIA, USA ZAN GOJCIC†, NVIDIA, Switzerland
This paper introduces a fast forward and inverse ray tracing method for particle-based scene representations, particularly 3D Gaussian Splats. The main idea is to encapsulate particles with bounding meshes to leverage fast ray-triangle intersections and shade batches of intersections in depth order. This approach allows for efficient handling of high-density overlapping particles and enables advanced techniques such as secondary ray effects, highly-distorted cameras, and stochastic sampling. The authors propose a specialized GPU-accelerated ray tracer that uses a k-buffer hits-based marching algorithm, bounding mesh proxies, and a backward pass for optimization. The method is evaluated on various benchmarks and applications, demonstrating speed and accuracy, as well as the ability to handle complex effects like depth of field, shadows, and mirrors. The contributions include a GPU-accelerated ray tracing algorithm for semi-transparent particles, an improved optimization pipeline, generalized Gaussian particle formulations, and applications in computer graphics and vision.This paper introduces a fast forward and inverse ray tracing method for particle-based scene representations, particularly 3D Gaussian Splats. The main idea is to encapsulate particles with bounding meshes to leverage fast ray-triangle intersections and shade batches of intersections in depth order. This approach allows for efficient handling of high-density overlapping particles and enables advanced techniques such as secondary ray effects, highly-distorted cameras, and stochastic sampling. The authors propose a specialized GPU-accelerated ray tracer that uses a k-buffer hits-based marching algorithm, bounding mesh proxies, and a backward pass for optimization. The method is evaluated on various benchmarks and applications, demonstrating speed and accuracy, as well as the ability to handle complex effects like depth of field, shadows, and mirrors. The contributions include a GPU-accelerated ray tracing algorithm for semi-transparent particles, an improved optimization pipeline, generalized Gaussian particle formulations, and applications in computer graphics and vision.