LRM-Zero: Training Large Reconstruction Models with Synthesized Data

LRM-Zero: Training Large Reconstruction Models with Synthesized Data

1 Nov 2024 | Desai Xie, Sai Bi, Zhixin Shu, Kai Zhang, Zexiang Xu, Yi Zhou, Sören Pirk, Arie Kaufman, Xin Sun, Hao Tan
LRM-Zero is a large reconstruction model trained entirely on synthesized 3D data, achieving high-quality sparse-view 3D reconstruction. The core of LRM-Zero is the Zeroverse dataset, a procedurally synthesized dataset created from simple primitive shapes with random texturing and augmentations. Unlike previous 3D datasets that are often captured or crafted by humans, Zeroverse ignores realistic global semantics but is rich in complex geometric and texture details. The Zeroverse dataset is used to train LRM-Zero, which can achieve high visual quality in reconstructing real-world objects, competitive with models trained on Objaverse. The paper analyzes the design choices of Zeroverse that contribute to LRM-Zero's capability and training stability. It also demonstrates that 3D reconstruction can be addressed without the semantics of real-world objects. The Zeroverse dataset and LRM-Zero model are available for further research. The paper also evaluates the performance of LRM-Zero on standard 3D reconstruction benchmarks and shows that it can generalize across different datasets, including realistic 3D data. The results show that LRM-Zero can achieve competitive performance with models trained on real-world data. The paper also discusses the limitations of using synthetic data, including scalability and semantic understanding. Overall, the paper demonstrates that synthetic data can be used to train large reconstruction models for 3D vision tasks.LRM-Zero is a large reconstruction model trained entirely on synthesized 3D data, achieving high-quality sparse-view 3D reconstruction. The core of LRM-Zero is the Zeroverse dataset, a procedurally synthesized dataset created from simple primitive shapes with random texturing and augmentations. Unlike previous 3D datasets that are often captured or crafted by humans, Zeroverse ignores realistic global semantics but is rich in complex geometric and texture details. The Zeroverse dataset is used to train LRM-Zero, which can achieve high visual quality in reconstructing real-world objects, competitive with models trained on Objaverse. The paper analyzes the design choices of Zeroverse that contribute to LRM-Zero's capability and training stability. It also demonstrates that 3D reconstruction can be addressed without the semantics of real-world objects. The Zeroverse dataset and LRM-Zero model are available for further research. The paper also evaluates the performance of LRM-Zero on standard 3D reconstruction benchmarks and shows that it can generalize across different datasets, including realistic 3D data. The results show that LRM-Zero can achieve competitive performance with models trained on real-world data. The paper also discusses the limitations of using synthetic data, including scalability and semantic understanding. Overall, the paper demonstrates that synthetic data can be used to train large reconstruction models for 3D vision tasks.
Reach us at info@futurestudyspace.com