ControlNet is a neural network architecture that adds spatial conditioning controls to large, pretrained text-to-image diffusion models. It allows users to add conditions like Canny edges, human pose, etc., to control the image generation process. The architecture uses "zero convolutions" to progressively grow parameters and prevent harmful noise from affecting the model. ControlNet is tested with various conditioning controls, including edges, depth, segmentation, and human pose, and is shown to be robust with small and large datasets. The model is trained to handle diverse input conditions and can be applied to various tasks, including depth-to-image and pose-to-image generation. ControlNet is implemented with Stable Diffusion and can be used with or without prompts. The model is also shown to be effective in generating images with multiple conditions and is compared to other baselines in user studies. The results show that ControlNet can generate high-quality images and is competitive with industrial models. The architecture is also shown to be transferable to other models in the stable diffusion community. The paper concludes that ControlNet is a robust and effective method for adding conditional control to text-to-image diffusion models.ControlNet is a neural network architecture that adds spatial conditioning controls to large, pretrained text-to-image diffusion models. It allows users to add conditions like Canny edges, human pose, etc., to control the image generation process. The architecture uses "zero convolutions" to progressively grow parameters and prevent harmful noise from affecting the model. ControlNet is tested with various conditioning controls, including edges, depth, segmentation, and human pose, and is shown to be robust with small and large datasets. The model is trained to handle diverse input conditions and can be applied to various tasks, including depth-to-image and pose-to-image generation. ControlNet is implemented with Stable Diffusion and can be used with or without prompts. The model is also shown to be effective in generating images with multiple conditions and is compared to other baselines in user studies. The results show that ControlNet can generate high-quality images and is competitive with industrial models. The architecture is also shown to be transferable to other models in the stable diffusion community. The paper concludes that ControlNet is a robust and effective method for adding conditional control to text-to-image diffusion models.