Mutual Information Neural Estimation

Mutual Information Neural Estimation

2018 | Mohamed Ishmael Belghazi 1 Aristide Baratin 1 2 Sai Rajeswar 1 Sherjil Ozair 1 Yoshua Bengio 1 3 4 R Devon Hjelm 1 4
The paper introduces the Mutual Information Neural Estimator (MINE), a method for estimating mutual information between high-dimensional continuous random variables. MINE is designed to be scalable, flexible, and trainable via backpropagation. It leverages dual representations of the Kullback-Leibler (KL) divergence to estimate mutual information, which can be used for minimization, maximization, and other applications. The paper provides a theoretical analysis of MINE's consistency and convergence properties, demonstrating its effectiveness in various settings. Empirical evaluations show that MINE outperforms non-parametric estimators and variational methods in estimating mutual information, especially in high-dimensional spaces. MINE is applied to improve generative models by mitigating mode collapse in GANs and enhancing inference in adversarially learned inference (ALI) models. Additionally, MINE is used to implement the Information Bottleneck method in a continuous setting, achieving better performance than existing methods.The paper introduces the Mutual Information Neural Estimator (MINE), a method for estimating mutual information between high-dimensional continuous random variables. MINE is designed to be scalable, flexible, and trainable via backpropagation. It leverages dual representations of the Kullback-Leibler (KL) divergence to estimate mutual information, which can be used for minimization, maximization, and other applications. The paper provides a theoretical analysis of MINE's consistency and convergence properties, demonstrating its effectiveness in various settings. Empirical evaluations show that MINE outperforms non-parametric estimators and variational methods in estimating mutual information, especially in high-dimensional spaces. MINE is applied to improve generative models by mitigating mode collapse in GANs and enhancing inference in adversarially learned inference (ALI) models. Additionally, MINE is used to implement the Information Bottleneck method in a continuous setting, achieving better performance than existing methods.
Reach us at info@study.space