2017 | Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, Kevin Murphy
The paper introduces the Deep Variational Information Bottleneck (Deep VIB), a variational approximation to the information bottleneck (IB) model proposed by Tishby et al. (1999). The Deep VIB method parameterizes the IB model using a neural network and leverages the reparameterization trick for efficient training. The authors show that models trained with the VIB objective outperform those trained with other forms of regularization in terms of generalization performance and robustness to adversarial attacks. The paper also discusses related work, including variational inference and information-theoretic objectives for deep neural networks, and presents experimental results on the MNIST and ImageNet datasets to demonstrate the effectiveness of the VIB method. The experiments show that VIB-trained models achieve better classification accuracy and are more robust to adversarial examples compared to deterministic models.The paper introduces the Deep Variational Information Bottleneck (Deep VIB), a variational approximation to the information bottleneck (IB) model proposed by Tishby et al. (1999). The Deep VIB method parameterizes the IB model using a neural network and leverages the reparameterization trick for efficient training. The authors show that models trained with the VIB objective outperform those trained with other forms of regularization in terms of generalization performance and robustness to adversarial attacks. The paper also discusses related work, including variational inference and information-theoretic objectives for deep neural networks, and presents experimental results on the MNIST and ImageNet datasets to demonstrate the effectiveness of the VIB method. The experiments show that VIB-trained models achieve better classification accuracy and are more robust to adversarial examples compared to deterministic models.