Panoptic segmentation is a task that unifies semantic segmentation (assigning class labels to pixels) and instance segmentation (detecting and segmenting object instances). It aims to generate a coherent, rich scene segmentation, an important step toward real-world vision systems. The proposed task requires a unified evaluation metric, the panoptic quality (PQ), which measures performance for all classes (stuff and things) in an interpretable and unified manner. Using PQ, the authors perform a rigorous study of both human and machine performance on three existing datasets, revealing insights about the task. The goal is to revive interest in a more unified view of image segmentation. The task format for panoptic segmentation is simple: each pixel of an image must be assigned a semantic label and an instance id. Pixels with the same label and id belong to the same object; for stuff labels, the instance id is ignored. This format has been used previously, especially by methods that produce non-overlapping instance segmentations. The task requires differentiating individual object instances, which poses a challenge for fully convolutional nets, and non-overlapping object segments, which presents a challenge for region-based methods. The panoptic segmentation metric, PQ, is a simple and informative metric that can be used to measure the performance for both stuff and things in a uniform manner. It involves two steps: segment matching and PQ computation given the matches. PQ is defined as the average IoU of matched segments, adjusted by a penalty for unmatched segments. It measures performance of all classes in a uniform way using a simple and interpretable formula. The authors study both human and machine performance on three popular segmentation datasets that have both stuff and things annotations. These include Cityscapes, ADE20k, and Mapillary Vistas. They also plan to extend their analysis to COCO, which has recently been annotated for stuff. The results on these datasets form a solid foundation for the study of both human and machine performance on panoptic segmentation. The authors also perform an initial study of machine performance for panoptic segmentation. They define a simple and likely suboptimal heuristic that combines the output of two independent systems for semantic and instance segmentation via a series of post-processing steps that merges their outputs. This heuristic establishes a baseline for PS and gives insights into the main algorithmic challenges it presents. The authors compare PQ to existing metrics for semantic and instance segmentation. They find that PQ is not a combination of semantic and instance segmentation metrics, but rather a unified metric that measures segmentation and recognition quality. They also discuss the future of panoptic segmentation, including the potential for new algorithms and research directions.Panoptic segmentation is a task that unifies semantic segmentation (assigning class labels to pixels) and instance segmentation (detecting and segmenting object instances). It aims to generate a coherent, rich scene segmentation, an important step toward real-world vision systems. The proposed task requires a unified evaluation metric, the panoptic quality (PQ), which measures performance for all classes (stuff and things) in an interpretable and unified manner. Using PQ, the authors perform a rigorous study of both human and machine performance on three existing datasets, revealing insights about the task. The goal is to revive interest in a more unified view of image segmentation. The task format for panoptic segmentation is simple: each pixel of an image must be assigned a semantic label and an instance id. Pixels with the same label and id belong to the same object; for stuff labels, the instance id is ignored. This format has been used previously, especially by methods that produce non-overlapping instance segmentations. The task requires differentiating individual object instances, which poses a challenge for fully convolutional nets, and non-overlapping object segments, which presents a challenge for region-based methods. The panoptic segmentation metric, PQ, is a simple and informative metric that can be used to measure the performance for both stuff and things in a uniform manner. It involves two steps: segment matching and PQ computation given the matches. PQ is defined as the average IoU of matched segments, adjusted by a penalty for unmatched segments. It measures performance of all classes in a uniform way using a simple and interpretable formula. The authors study both human and machine performance on three popular segmentation datasets that have both stuff and things annotations. These include Cityscapes, ADE20k, and Mapillary Vistas. They also plan to extend their analysis to COCO, which has recently been annotated for stuff. The results on these datasets form a solid foundation for the study of both human and machine performance on panoptic segmentation. The authors also perform an initial study of machine performance for panoptic segmentation. They define a simple and likely suboptimal heuristic that combines the output of two independent systems for semantic and instance segmentation via a series of post-processing steps that merges their outputs. This heuristic establishes a baseline for PS and gives insights into the main algorithmic challenges it presents. The authors compare PQ to existing metrics for semantic and instance segmentation. They find that PQ is not a combination of semantic and instance segmentation metrics, but rather a unified metric that measures segmentation and recognition quality. They also discuss the future of panoptic segmentation, including the potential for new algorithms and research directions.