Generative Modeling by Estimating Gradients of the Data Distribution

Generative Modeling by Estimating Gradients of the Data Distribution

10 Oct 2020 | Yang Song, Stefano Ermon
This paper introduces a new generative model based on estimating gradients of the data distribution using score matching and Langevin dynamics. The model, called Noise Conditional Score Networks (NCSNs), addresses the challenges of generating samples from complex data distributions by perturbing data with Gaussian noise and jointly estimating scores for all noise levels. The key idea is to train a single score network conditioned on noise levels, allowing for flexible model architectures and avoiding adversarial training or MCMC sampling during training. The model uses an annealed Langevin dynamics approach for sampling, which gradually reduces noise levels to generate samples that are comparable to those from GANs and likelihood-based models. The proposed method achieves state-of-the-art results on the CIFAR-10 dataset, with an inception score of 8.87 and a FID score of 25.32. The model also demonstrates effective image representations through image inpainting experiments. The approach is scalable, efficient, and provides a principled learning objective for model comparison. The paper also discusses the challenges of score-based generative modeling, including the manifold hypothesis and low data density regions, and proposes solutions to overcome these issues. The method is validated on multiple image datasets, showing its effectiveness in generating high-quality samples and learning meaningful representations.This paper introduces a new generative model based on estimating gradients of the data distribution using score matching and Langevin dynamics. The model, called Noise Conditional Score Networks (NCSNs), addresses the challenges of generating samples from complex data distributions by perturbing data with Gaussian noise and jointly estimating scores for all noise levels. The key idea is to train a single score network conditioned on noise levels, allowing for flexible model architectures and avoiding adversarial training or MCMC sampling during training. The model uses an annealed Langevin dynamics approach for sampling, which gradually reduces noise levels to generate samples that are comparable to those from GANs and likelihood-based models. The proposed method achieves state-of-the-art results on the CIFAR-10 dataset, with an inception score of 8.87 and a FID score of 25.32. The model also demonstrates effective image representations through image inpainting experiments. The approach is scalable, efficient, and provides a principled learning objective for model comparison. The paper also discusses the challenges of score-based generative modeling, including the manifold hypothesis and low data density regions, and proposes solutions to overcome these issues. The method is validated on multiple image datasets, showing its effectiveness in generating high-quality samples and learning meaningful representations.
Reach us at info@study.space
[slides] Generative Modeling by Estimating Gradients of the Data Distribution | StudySpace