The paper introduces Matterport3D, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400 RGB-D images of 90 building-scale scenes. The dataset includes surface reconstructions, camera poses, and 2D and 3D semantic segmentations. It provides a comprehensive and diverse set of views over entire buildings, enabling various supervised and self-supervised computer vision tasks. The dataset's unique properties, such as precise global alignment, panoramic and comprehensive viewpoint sampling, and high-quality depth and color images, enable advancements in keypoint matching, view overlap prediction, normal prediction from color, semantic segmentation, and region classification. The paper also discusses the background and related work, the acquisition process of the dataset, and the semantic annotation process. Finally, it presents experimental results demonstrating the effectiveness of the Matterport3D dataset in improving performance on these tasks.The paper introduces Matterport3D, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400 RGB-D images of 90 building-scale scenes. The dataset includes surface reconstructions, camera poses, and 2D and 3D semantic segmentations. It provides a comprehensive and diverse set of views over entire buildings, enabling various supervised and self-supervised computer vision tasks. The dataset's unique properties, such as precise global alignment, panoramic and comprehensive viewpoint sampling, and high-quality depth and color images, enable advancements in keypoint matching, view overlap prediction, normal prediction from color, semantic segmentation, and region classification. The paper also discusses the background and related work, the acquisition process of the dataset, and the semantic annotation process. Finally, it presents experimental results demonstrating the effectiveness of the Matterport3D dataset in improving performance on these tasks.