6 Nov 2017 | Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky
This paper revisits the fast stylization method introduced by Ulyanov et al. (2016) and shows that replacing batch normalization with instance normalization in the generator architecture significantly improves the quality of generated images. The change is simple: swap batch normalization with instance normalization, and apply instance normalization both during training and testing. The resulting method can be used to train high-performance architectures for real-time image generation. The key idea is that instance normalization allows to remove instance-specific contrast information from the content image, which simplifies generation. This results in vastly improved images compared to the original method.
The method of Gatys et al. (2016) produces remarkably good results but is computationally inefficient. It requires iterative optimization to match desired statistics. In contrast, the fast stylization method of Ulyanov et al. (2016) uses a feed-forward generator network to generate stylized images in a single pass. However, the results were not as good as the slower optimization-based method. The authors show that replacing batch normalization with instance normalization in the generator architecture leads to much improved results. The results are of comparable quality as the slow optimization method but can be obtained in real time on standard GPU hardware.
The generator network is a convolutional neural network learned from examples. The loss function uses a pre-trained CNN to extract features from the style and content images and compares their statistics. The authors found that learning the generator from too many training examples yields poorer qualitative results. They conjectured that the training objective was too hard to learn for a standard neural network architecture.
The key difference between instance normalization and batch normalization is that instance normalization applies normalization to a single image instead of a whole batch of images. The authors propose to replace batch normalization with instance normalization in the generator network. This prevents instance-specific mean and covariance shift, simplifying the learning process. Instance normalization is also applied at test time, unlike batch normalization.
Experiments show that replacing batch normalization with instance normalization significantly improves the performance of the generator networks. Both architectures benefit from instance normalization, with the residuals architecture of Johnson et al. (2016) being somewhat more efficient and easy to use. The results show that instance normalization can dramatically improve the performance of certain deep neural networks for image generation.This paper revisits the fast stylization method introduced by Ulyanov et al. (2016) and shows that replacing batch normalization with instance normalization in the generator architecture significantly improves the quality of generated images. The change is simple: swap batch normalization with instance normalization, and apply instance normalization both during training and testing. The resulting method can be used to train high-performance architectures for real-time image generation. The key idea is that instance normalization allows to remove instance-specific contrast information from the content image, which simplifies generation. This results in vastly improved images compared to the original method.
The method of Gatys et al. (2016) produces remarkably good results but is computationally inefficient. It requires iterative optimization to match desired statistics. In contrast, the fast stylization method of Ulyanov et al. (2016) uses a feed-forward generator network to generate stylized images in a single pass. However, the results were not as good as the slower optimization-based method. The authors show that replacing batch normalization with instance normalization in the generator architecture leads to much improved results. The results are of comparable quality as the slow optimization method but can be obtained in real time on standard GPU hardware.
The generator network is a convolutional neural network learned from examples. The loss function uses a pre-trained CNN to extract features from the style and content images and compares their statistics. The authors found that learning the generator from too many training examples yields poorer qualitative results. They conjectured that the training objective was too hard to learn for a standard neural network architecture.
The key difference between instance normalization and batch normalization is that instance normalization applies normalization to a single image instead of a whole batch of images. The authors propose to replace batch normalization with instance normalization in the generator network. This prevents instance-specific mean and covariance shift, simplifying the learning process. Instance normalization is also applied at test time, unlike batch normalization.
Experiments show that replacing batch normalization with instance normalization significantly improves the performance of the generator networks. Both architectures benefit from instance normalization, with the residuals architecture of Johnson et al. (2016) being somewhat more efficient and easy to use. The results show that instance normalization can dramatically improve the performance of certain deep neural networks for image generation.