Rényi Divergence and Kullback-Leibler Divergence

Rényi Divergence and Kullback-Leibler Divergence

24 Apr 2014 | Tim van Erven, Peter Harremoës, Member, IEEE
The paper reviews and extends the properties of Rényi divergence, which is related to Rényi entropy and Kullback-Leibler divergence. It covers convexity, continuity, limits of σ-algebras, and the relationship between the special order 0 and the Gaussian dichotomy and contiguity. The paper also generalizes the Pythagorean inequality to orders different from 1 and extends the equivalence between channel capacity and minimax redundancy to continuous channel inputs for all orders. Additionally, it discusses the connection to Chernoff information in hypothesis testing and the total variation topology. The paper includes counterexamples to properties that hold for other divergences but not for Rényi divergence.The paper reviews and extends the properties of Rényi divergence, which is related to Rényi entropy and Kullback-Leibler divergence. It covers convexity, continuity, limits of σ-algebras, and the relationship between the special order 0 and the Gaussian dichotomy and contiguity. The paper also generalizes the Pythagorean inequality to orders different from 1 and extends the equivalence between channel capacity and minimax redundancy to continuous channel inputs for all orders. Additionally, it discusses the connection to Chernoff information in hypothesis testing and the total variation topology. The paper includes counterexamples to properties that hold for other divergences but not for Rényi divergence.
Reach us at info@study.space
[slides and audio] R%C3%A9nyi Divergence and Kullback-Leibler Divergence