This paper proposes a simple latent diffusion approach for panoptic segmentation and mask inpainting. The method builds upon Stable Diffusion and introduces a latent diffusion model for segmentation, resulting in a simple architecture that avoids complex modules and loss functions. The approach consists of two stages: (1) training a shallow autoencoder to project segmentation masks to latent space; (2) training a diffusion model to allow image-conditioned sampling in latent space. This generative approach enables mask completion or inpainting. The model is validated on COCO and ADE20k, achieving strong segmentation results. The model is also adaptable to multi-task learning by introducing learnable task embeddings. The code and models will be made available. The paper discusses related work, including panoptic segmentation, general-purpose frameworks, and denoising diffusion models. The method is evaluated on various tasks, including segmentation, mask inpainting, and multi-task learning. The results show that the proposed approach achieves strong performance on panoptic segmentation and mask inpainting, while also being adaptable to multiple tasks. The paper concludes that the proposed approach is a simple yet powerful method for panoptic segmentation and mask inpainting, with potential for further improvements in accuracy and sampling speed.This paper proposes a simple latent diffusion approach for panoptic segmentation and mask inpainting. The method builds upon Stable Diffusion and introduces a latent diffusion model for segmentation, resulting in a simple architecture that avoids complex modules and loss functions. The approach consists of two stages: (1) training a shallow autoencoder to project segmentation masks to latent space; (2) training a diffusion model to allow image-conditioned sampling in latent space. This generative approach enables mask completion or inpainting. The model is validated on COCO and ADE20k, achieving strong segmentation results. The model is also adaptable to multi-task learning by introducing learnable task embeddings. The code and models will be made available. The paper discusses related work, including panoptic segmentation, general-purpose frameworks, and denoising diffusion models. The method is evaluated on various tasks, including segmentation, mask inpainting, and multi-task learning. The results show that the proposed approach achieves strong performance on panoptic segmentation and mask inpainting, while also being adaptable to multiple tasks. The paper concludes that the proposed approach is a simple yet powerful method for panoptic segmentation and mask inpainting, with potential for further improvements in accuracy and sampling speed.