This paper explores the impact of different observation spaces on robot learning, focusing on RGB, RGB-D, and point cloud modalities. The authors introduce OBSBench, a benchmark comprising two simulators and 125 tasks, along with standardized pipelines for various encoders and policy baselines. Extensive experiments on diverse contact-rich manipulation tasks reveal that point cloud-based methods consistently outperform their RGB and RGB-D counterparts, both in terms of success rate and robustness. Point cloud methods show superior performance in zero-shot generalization to camera views and visual changes, and they are more robust to variations in camera viewpoints and lighting conditions. The study also highlights the importance of explicit 3D representations and the benefits of incorporating both color and coordinate information. Additionally, the paper discusses the influence of design choices, such as post-sampling and the use of pre-trained visual representations (PVRs), on the performance of point cloud methods. The findings suggest that point clouds are a valuable observation modality for intricate robotic tasks, and further research should explore dynamic sampling techniques and multi-modal integration.This paper explores the impact of different observation spaces on robot learning, focusing on RGB, RGB-D, and point cloud modalities. The authors introduce OBSBench, a benchmark comprising two simulators and 125 tasks, along with standardized pipelines for various encoders and policy baselines. Extensive experiments on diverse contact-rich manipulation tasks reveal that point cloud-based methods consistently outperform their RGB and RGB-D counterparts, both in terms of success rate and robustness. Point cloud methods show superior performance in zero-shot generalization to camera views and visual changes, and they are more robust to variations in camera viewpoints and lighting conditions. The study also highlights the importance of explicit 3D representations and the benefits of incorporating both color and coordinate information. Additionally, the paper discusses the influence of design choices, such as post-sampling and the use of pre-trained visual representations (PVRs), on the performance of point cloud methods. The findings suggest that point clouds are a valuable observation modality for intricate robotic tasks, and further research should explore dynamic sampling techniques and multi-modal integration.