Estimating Mutual Information

Estimating Mutual Information

February 2, 2008 | Alexander Kraskov, Harald St"ogbauer, and Peter Grassberger
This paper presents two improved estimators for mutual information (MI) between random variables based on k-nearest neighbor (k-NN) distances, which are more efficient and accurate than traditional binning-based methods. The estimators are data-efficient, adaptive, and have minimal bias. They are particularly effective for independent distributions, where the estimator vanishes (up to statistical fluctuations) when the joint distribution factors into the product of marginal distributions. The methods are also applicable to redundancies between more than two variables. The first estimator, $ I^{(1)}(X,Y) $, uses the k-th nearest neighbor distances in the joint space $ Z = (X,Y) $, while the second estimator, $ I^{(2)}(X,Y) $, uses distances in the marginal subspaces $ X $ and $ Y $. Both estimators are derived from entropy estimates based on k-NN distances and are shown to be exact for independent variables. The first estimator is given by: $$ I^{(1)}(X,Y) = \psi(k) - \langle \psi(n_{x}+1) + \psi(n_{y}+1) \rangle + \psi(N) $$ The second estimator is: $$ I^{(2)}(X,Y) = \psi(k) - 1/k - \langle \psi(n_{x}) + \psi(n_{y}) \rangle + \psi(N) $$ where $ n_{x}(i) $ and $ n_{y}(i) $ are the number of points within a certain distance of the i-th point in the marginal subspaces $ X $ and $ Y $, respectively. The estimators are tested on various distributions, including Gaussian, gamma-exponential, and ordered Weinman exponential distributions. They are shown to be accurate and efficient, with systematic errors that decrease as the number of data points increases. The estimators are also compared with existing methods, such as the Darbellay-Vajda algorithm, and are found to be competitive in terms of accuracy and efficiency. The paper concludes that the proposed estimators are effective for estimating mutual information and redundancies between random variables, and that they are particularly useful for assessing the independence of components obtained from independent component analysis (ICA), improving ICA, and estimating the reliability of blind source separation.This paper presents two improved estimators for mutual information (MI) between random variables based on k-nearest neighbor (k-NN) distances, which are more efficient and accurate than traditional binning-based methods. The estimators are data-efficient, adaptive, and have minimal bias. They are particularly effective for independent distributions, where the estimator vanishes (up to statistical fluctuations) when the joint distribution factors into the product of marginal distributions. The methods are also applicable to redundancies between more than two variables. The first estimator, $ I^{(1)}(X,Y) $, uses the k-th nearest neighbor distances in the joint space $ Z = (X,Y) $, while the second estimator, $ I^{(2)}(X,Y) $, uses distances in the marginal subspaces $ X $ and $ Y $. Both estimators are derived from entropy estimates based on k-NN distances and are shown to be exact for independent variables. The first estimator is given by: $$ I^{(1)}(X,Y) = \psi(k) - \langle \psi(n_{x}+1) + \psi(n_{y}+1) \rangle + \psi(N) $$ The second estimator is: $$ I^{(2)}(X,Y) = \psi(k) - 1/k - \langle \psi(n_{x}) + \psi(n_{y}) \rangle + \psi(N) $$ where $ n_{x}(i) $ and $ n_{y}(i) $ are the number of points within a certain distance of the i-th point in the marginal subspaces $ X $ and $ Y $, respectively. The estimators are tested on various distributions, including Gaussian, gamma-exponential, and ordered Weinman exponential distributions. They are shown to be accurate and efficient, with systematic errors that decrease as the number of data points increases. The estimators are also compared with existing methods, such as the Darbellay-Vajda algorithm, and are found to be competitive in terms of accuracy and efficiency. The paper concludes that the proposed estimators are effective for estimating mutual information and redundancies between random variables, and that they are particularly useful for assessing the independence of components obtained from independent component analysis (ICA), improving ICA, and estimating the reliability of blind source separation.
Reach us at info@study.space
[slides and audio] Estimating mutual information.