Neural 3D Mesh Renderer

Neural 3D Mesh Renderer

20 Nov 2017 | Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada
This paper proposes a neural renderer for 3D mesh rendering that enables the integration of rendering into neural networks. The key contribution is the development of an approximate gradient for rasterization, which allows gradients to flow through the rendering process, enabling end-to-end training. The proposed renderer supports single-image 3D mesh reconstruction with silhouette image supervision and gradient-based 3D mesh editing, such as 2D-to-3D style transfer and 3D DeepDream, using 2D supervision for the first time. The renderer is capable of generating gradients for texture, lighting, and camera parameters, making it applicable to a wide range of problems. The method is evaluated on two applications: single-image 3D mesh reconstruction and gradient-based 3D mesh editing. The results show that the mesh-based approach outperforms the voxel-based approach in terms of visual appeal and voxel IoU metric. The applications demonstrate the potential of integrating mesh renderers into neural networks and the effectiveness of the proposed renderer. The applications of the renderer are not limited to those presented in this paper, and other problems can be solved by incorporating the module into other systems.This paper proposes a neural renderer for 3D mesh rendering that enables the integration of rendering into neural networks. The key contribution is the development of an approximate gradient for rasterization, which allows gradients to flow through the rendering process, enabling end-to-end training. The proposed renderer supports single-image 3D mesh reconstruction with silhouette image supervision and gradient-based 3D mesh editing, such as 2D-to-3D style transfer and 3D DeepDream, using 2D supervision for the first time. The renderer is capable of generating gradients for texture, lighting, and camera parameters, making it applicable to a wide range of problems. The method is evaluated on two applications: single-image 3D mesh reconstruction and gradient-based 3D mesh editing. The results show that the mesh-based approach outperforms the voxel-based approach in terms of visual appeal and voxel IoU metric. The applications demonstrate the potential of integrating mesh renderers into neural networks and the effectiveness of the proposed renderer. The applications of the renderer are not limited to those presented in this paper, and other problems can be solved by incorporating the module into other systems.
Reach us at info@study.space
[slides and audio] Neural 3D Mesh Renderer