The paper introduces COCO-Stuff, a new dataset that extends the COCO 2017 dataset with pixel-level annotations for 91 stuff classes. The COCO dataset originally includes annotations for 80 thing classes. COCO-Stuff adds detailed stuff annotations, enabling the study of interactions between things and stuff in complex scenes. The dataset includes 164,000 images, with annotations for both thing and stuff classes. The paper presents an efficient annotation protocol that leverages existing thing annotations and superpixels to achieve high-quality, efficient stuff annotations. The dataset is used to analyze the importance of stuff and thing classes in terms of their surface coverage and frequency in image captions, spatial relations between stuff and things, and the performance of semantic segmentation methods on both classes. The results show that stuff is not generally easier to segment than things, and that COCO-Stuff provides a rich set of annotations that support detailed analysis of scene understanding. The paper also compares COCO-Stuff with other datasets, showing that it has the largest number of images and a more diverse set of stuff and thing classes. The dataset includes natural language captions, further supporting rich scene understanding. The paper concludes that COCO-Stuff is a valuable resource for studying the relationships between things and stuff in complex scenes.The paper introduces COCO-Stuff, a new dataset that extends the COCO 2017 dataset with pixel-level annotations for 91 stuff classes. The COCO dataset originally includes annotations for 80 thing classes. COCO-Stuff adds detailed stuff annotations, enabling the study of interactions between things and stuff in complex scenes. The dataset includes 164,000 images, with annotations for both thing and stuff classes. The paper presents an efficient annotation protocol that leverages existing thing annotations and superpixels to achieve high-quality, efficient stuff annotations. The dataset is used to analyze the importance of stuff and thing classes in terms of their surface coverage and frequency in image captions, spatial relations between stuff and things, and the performance of semantic segmentation methods on both classes. The results show that stuff is not generally easier to segment than things, and that COCO-Stuff provides a rich set of annotations that support detailed analysis of scene understanding. The paper also compares COCO-Stuff with other datasets, showing that it has the largest number of images and a more diverse set of stuff and thing classes. The dataset includes natural language captions, further supporting rich scene understanding. The paper concludes that COCO-Stuff is a valuable resource for studying the relationships between things and stuff in complex scenes.