Generative Modeling by Estimating Gradients of the Data Distribution

Generative Modeling by Estimating Gradients of the Data Distribution

10 Oct 2020 | Yang Song, Stefano Ermon
The paper introduces a new generative model that uses Langevin dynamics to produce samples, with gradients of the data distribution estimated using score matching. To address the challenges of estimating gradients on low-dimensional manifolds, the authors perturb the data with Gaussian noise at different levels and jointly estimate the corresponding scores. They propose an annealed Langevin dynamics sampling method, where the initial samples are drawn from a high-noise distribution and gradually annealed to the original data distribution. This approach allows for flexible model architectures, does not require sampling during training, and provides a principled learning objective for model comparisons. The models achieve state-of-the-art inception scores on CIFAR-10 and competitive FID scores, demonstrating effective representation learning through image inpainting experiments.The paper introduces a new generative model that uses Langevin dynamics to produce samples, with gradients of the data distribution estimated using score matching. To address the challenges of estimating gradients on low-dimensional manifolds, the authors perturb the data with Gaussian noise at different levels and jointly estimate the corresponding scores. They propose an annealed Langevin dynamics sampling method, where the initial samples are drawn from a high-noise distribution and gradually annealed to the original data distribution. This approach allows for flexible model architectures, does not require sampling during training, and provides a principled learning objective for model comparisons. The models achieve state-of-the-art inception scores on CIFAR-10 and competitive FID scores, demonstrating effective representation learning through image inpainting experiments.
Reach us at info@study.space
Understanding Generative Modeling by Estimating Gradients of the Data Distribution