IMPORTANCE WEIGHTED AUTOENCODERS

IMPORTANCE WEIGHTED AUTOENCODERS

7 Nov 2016 | Yuri Burda, Roger Grosse & Ruslan Salakhutdinov
The Importance Weighted Autoencoder (IWAE) is a generative model that improves upon the Variational Autoencoder (VAE) by using a tighter log-likelihood lower bound derived from importance weighting. The VAE, which pairs a generative network with a recognition network, makes strong assumptions about posterior inference, such as factorial independence and approximability via nonlinear regression. These assumptions can lead to overly simplified representations. The IWAE addresses this by using multiple samples to approximate the posterior, allowing it to model more complex posteriors. Empirically, IWAEs learn richer latent space representations and achieve better performance on density estimation benchmarks compared to VAEs. The IWAE is trained to maximize a tighter lower bound on the log-likelihood, which is derived from importance weighting. This approach allows the model to better approximate the true log-likelihood and improves generative performance. The training procedure involves using the reparameterization trick to derive a low-variance update rule, and the use of multiple samples helps reduce variance in the gradient estimates. The IWAE also benefits from using normalized importance weights, which improve the efficiency of the training process. Experimental results show that IWAEs outperform VAEs in terms of log-likelihood and latent space representation quality, particularly when using multiple samples. The models were evaluated on benchmark datasets such as MNIST and Omniglot, where IWAEs achieved higher log-likelihoods and more effective latent representations. The study also highlights the importance of using multiple samples in posterior inference to avoid over-simplification and improve model performance.The Importance Weighted Autoencoder (IWAE) is a generative model that improves upon the Variational Autoencoder (VAE) by using a tighter log-likelihood lower bound derived from importance weighting. The VAE, which pairs a generative network with a recognition network, makes strong assumptions about posterior inference, such as factorial independence and approximability via nonlinear regression. These assumptions can lead to overly simplified representations. The IWAE addresses this by using multiple samples to approximate the posterior, allowing it to model more complex posteriors. Empirically, IWAEs learn richer latent space representations and achieve better performance on density estimation benchmarks compared to VAEs. The IWAE is trained to maximize a tighter lower bound on the log-likelihood, which is derived from importance weighting. This approach allows the model to better approximate the true log-likelihood and improves generative performance. The training procedure involves using the reparameterization trick to derive a low-variance update rule, and the use of multiple samples helps reduce variance in the gradient estimates. The IWAE also benefits from using normalized importance weights, which improve the efficiency of the training process. Experimental results show that IWAEs outperform VAEs in terms of log-likelihood and latent space representation quality, particularly when using multiple samples. The models were evaluated on benchmark datasets such as MNIST and Omniglot, where IWAEs achieved higher log-likelihoods and more effective latent representations. The study also highlights the importance of using multiple samples in posterior inference to avoid over-simplification and improve model performance.
Reach us at info@study.space
Understanding Importance Weighted Autoencoders