15 Dec 2018 | Guilin Liu Fitsum A. Reda Kevin J. Shih Ting-Chun Wang Andrew Tao Bryan Catanzaro
This paper proposes a partial convolutional network for image inpainting, which addresses the issue of artifacts caused by traditional convolutional networks that condition on both valid pixels and substitute values in masked holes. The proposed method uses partial convolutions, where the convolution is masked and renormalized to be conditioned only on valid pixels. Additionally, the method includes an automatic mask update step during the forward pass to refine the mask for the next layer. The model outperforms other methods for irregular masks, as demonstrated through qualitative and quantitative comparisons.
The paper discusses the limitations of previous approaches, including their focus on rectangular holes and reliance on post-processing. The proposed method is designed to handle irregular hole patterns and produce semantically meaningful predictions without additional post-processing. The model uses a partial convolutional layer, which allows the output to depend only on the non-hole regions at every layer. The automatic mask update step ensures that the mask is updated based on the effective region of the convolution, leading to more accurate inpainting results.
The paper also presents a detailed discussion of the network architecture, loss functions, and training process. The model uses a UNet-like architecture with partial convolutional layers and nearest neighbor up-sampling. The loss functions include per-pixel reconstruction accuracy, perceptual loss, style loss, and total variation loss. The model is trained on a large irregular mask dataset, which includes masks of varying sizes and shapes.
The experiments show that the proposed method outperforms other methods in terms of inpainting quality, especially for irregular masks. The model is also evaluated through a user study, where it was found to be preferred over other methods in most cases. The paper also extends the framework to image super-resolution tasks, demonstrating its versatility.
The paper concludes that the proposed method achieves state-of-the-art results in image inpainting, particularly for irregular masks, and provides a new approach to handling irregular hole patterns in image inpainting.This paper proposes a partial convolutional network for image inpainting, which addresses the issue of artifacts caused by traditional convolutional networks that condition on both valid pixels and substitute values in masked holes. The proposed method uses partial convolutions, where the convolution is masked and renormalized to be conditioned only on valid pixels. Additionally, the method includes an automatic mask update step during the forward pass to refine the mask for the next layer. The model outperforms other methods for irregular masks, as demonstrated through qualitative and quantitative comparisons.
The paper discusses the limitations of previous approaches, including their focus on rectangular holes and reliance on post-processing. The proposed method is designed to handle irregular hole patterns and produce semantically meaningful predictions without additional post-processing. The model uses a partial convolutional layer, which allows the output to depend only on the non-hole regions at every layer. The automatic mask update step ensures that the mask is updated based on the effective region of the convolution, leading to more accurate inpainting results.
The paper also presents a detailed discussion of the network architecture, loss functions, and training process. The model uses a UNet-like architecture with partial convolutional layers and nearest neighbor up-sampling. The loss functions include per-pixel reconstruction accuracy, perceptual loss, style loss, and total variation loss. The model is trained on a large irregular mask dataset, which includes masks of varying sizes and shapes.
The experiments show that the proposed method outperforms other methods in terms of inpainting quality, especially for irregular masks. The model is also evaluated through a user study, where it was found to be preferred over other methods in most cases. The paper also extends the framework to image super-resolution tasks, demonstrating its versatility.
The paper concludes that the proposed method achieves state-of-the-art results in image inpainting, particularly for irregular masks, and provides a new approach to handling irregular hole patterns in image inpainting.