**SpatialVLM: Enhancing Vision-Language Models with Spatial Reasoning** **Authors:** Boyuan Chen, Zhuo Xu, Sean Kirmani, Danny Driess, Pete Florence, Brian Ichter, Dorsa Sadigh, Leonidas Guibas, Fei Xia **Institution:** Google DeepMind, Google Research **Abstract:** SpatialVLM is a novel framework designed to enhance the spatial reasoning capabilities of Vision-Language Models (VLMs). The authors address the limitation of current VLMs in 3D spatial reasoning by training them on synthetic data generated from real-world images. This synthetic data, which includes metric depth estimation and object-centric captioning, is used to train VLMs on a large scale, resulting in significant improvements in both qualitative and quantitative spatial reasoning tasks. The framework is evaluated on various benchmarks, demonstrating superior performance compared to state-of-the-art VLMs like GPT-4V. Additionally, the enhanced spatial reasoning capabilities of SpatialVLM are leveraged for novel applications such as chain-of-thought spatial reasoning and robotics, where it can serve as a dense reward annotator and perform complex spatial reasoning tasks. **Key Contributions:** 1. **Enhanced Spatial Reasoning:** SpatialVLM significantly improves VLMs' ability to reason about spatial relationships, including qualitative and quantitative spatial reasoning. 2. **Large-Scale Synthetic Data:** The framework generates a large dataset of 2 billion spatial reasoning questions and answers, featuring diverse object descriptions and question types. 3. **Robotic Applications:** SpatialVLM enables VLMs to perform complex spatial reasoning tasks in robotics, such as reward annotation and success detection. **Methods:** - **Data Generation:** The system uses off-the-shelf computer vision models to extract object-centric contexts from 2D images, lifts these contexts into 3D point clouds, and generates spatial reasoning questions and answers. - **Training:** VLMs are trained on a mixture of captioning, VQA, and spatial reasoning data, with a focus on direct spatial reasoning tasks. - **Evaluation:** Experiments show that SpatialVLM outperforms baselines in both qualitative and quantitative spatial reasoning tasks, demonstrating its effectiveness in real-world applications. **Results:** - **Qualitative Spatial Reasoning:** SpatialVLM achieves higher accuracy in answering qualitative spatial reasoning questions compared to baselines. - **Quantitative Spatial Reasoning:** The model performs better in estimating distances and sizes, with more accurate and consistent answers. - **Robotic Applications:** SpatialVLM can be used as a dense reward annotator in robotics tasks, demonstrating its practical utility. **Conclusion:** SpatialVLM addresses the challenge of enhancing VLMs' spatial reasoning capabilities by generating and training on large-scale synthetic data. The framework not only improves VLMs' performance in spatial reasoning tasks but also opens up new applications in robotics and other fields.**SpatialVLM: Enhancing Vision-Language Models with Spatial Reasoning** **Authors:** Boyuan Chen, Zhuo Xu, Sean Kirmani, Danny Driess, Pete Florence, Brian Ichter, Dorsa Sadigh, Leonidas Guibas, Fei Xia **Institution:** Google DeepMind, Google Research **Abstract:** SpatialVLM is a novel framework designed to enhance the spatial reasoning capabilities of Vision-Language Models (VLMs). The authors address the limitation of current VLMs in 3D spatial reasoning by training them on synthetic data generated from real-world images. This synthetic data, which includes metric depth estimation and object-centric captioning, is used to train VLMs on a large scale, resulting in significant improvements in both qualitative and quantitative spatial reasoning tasks. The framework is evaluated on various benchmarks, demonstrating superior performance compared to state-of-the-art VLMs like GPT-4V. Additionally, the enhanced spatial reasoning capabilities of SpatialVLM are leveraged for novel applications such as chain-of-thought spatial reasoning and robotics, where it can serve as a dense reward annotator and perform complex spatial reasoning tasks. **Key Contributions:** 1. **Enhanced Spatial Reasoning:** SpatialVLM significantly improves VLMs' ability to reason about spatial relationships, including qualitative and quantitative spatial reasoning. 2. **Large-Scale Synthetic Data:** The framework generates a large dataset of 2 billion spatial reasoning questions and answers, featuring diverse object descriptions and question types. 3. **Robotic Applications:** SpatialVLM enables VLMs to perform complex spatial reasoning tasks in robotics, such as reward annotation and success detection. **Methods:** - **Data Generation:** The system uses off-the-shelf computer vision models to extract object-centric contexts from 2D images, lifts these contexts into 3D point clouds, and generates spatial reasoning questions and answers. - **Training:** VLMs are trained on a mixture of captioning, VQA, and spatial reasoning data, with a focus on direct spatial reasoning tasks. - **Evaluation:** Experiments show that SpatialVLM outperforms baselines in both qualitative and quantitative spatial reasoning tasks, demonstrating its effectiveness in real-world applications. **Results:** - **Qualitative Spatial Reasoning:** SpatialVLM achieves higher accuracy in answering qualitative spatial reasoning questions compared to baselines. - **Quantitative Spatial Reasoning:** The model performs better in estimating distances and sizes, with more accurate and consistent answers. - **Robotic Applications:** SpatialVLM can be used as a dense reward annotator in robotics tasks, demonstrating its practical utility. **Conclusion:** SpatialVLM addresses the challenge of enhancing VLMs' spatial reasoning capabilities by generating and training on large-scale synthetic data. The framework not only improves VLMs' performance in spatial reasoning tasks but also opens up new applications in robotics and other fields.