This paper explores the inner workings of Deep Neural Networks (DNNs) through the lens of Information Theory, focusing on the Information Plane, which visualizes the Mutual Information values between input and output variables across layers. The authors build on previous work by Tishby and Zaslavsky (2015), who proposed that DNNs optimize the Information Bottleneck (IB) tradeoff between compression and prediction. The main findings include: 1. **Training Dynamics**: Most training epochs are spent on compressing the input representation rather than fitting the training labels. The compression phase begins when training errors become small and SGD transitions from a fast drift to a stochastic relaxation phase. 2. **Layer Convergence**: converged layers lie close to or on the IB theoretical bound, satisfying the IB self-consistent equations. This indicates that the hidden layers effectively compress the input while preserving relevant information. 3. **Computational Benefits**: Adding more hidden layers significantly reduces training time by accelerating the compression phase, which is crucial for computational efficiency. 4. **Critical Points**: Hidden layers tend to converge to critical points on the IB curve, which can be explained by critical slowing down during stochastic relaxation. The paper also discusses the implications of these findings for understanding and improving DNNs, suggesting that the compression phase and convergence to the IB bound are key to the success of Deep Learning. The authors conclude that DNNs effectively learn efficient representations that are approximate minimal sufficient statistics in the IB sense, and that the IB framework can provide new insights and algorithms for training DNNs.This paper explores the inner workings of Deep Neural Networks (DNNs) through the lens of Information Theory, focusing on the Information Plane, which visualizes the Mutual Information values between input and output variables across layers. The authors build on previous work by Tishby and Zaslavsky (2015), who proposed that DNNs optimize the Information Bottleneck (IB) tradeoff between compression and prediction. The main findings include: 1. **Training Dynamics**: Most training epochs are spent on compressing the input representation rather than fitting the training labels. The compression phase begins when training errors become small and SGD transitions from a fast drift to a stochastic relaxation phase. 2. **Layer Convergence**: converged layers lie close to or on the IB theoretical bound, satisfying the IB self-consistent equations. This indicates that the hidden layers effectively compress the input while preserving relevant information. 3. **Computational Benefits**: Adding more hidden layers significantly reduces training time by accelerating the compression phase, which is crucial for computational efficiency. 4. **Critical Points**: Hidden layers tend to converge to critical points on the IB curve, which can be explained by critical slowing down during stochastic relaxation. The paper also discusses the implications of these findings for understanding and improving DNNs, suggesting that the compression phase and convergence to the IB bound are key to the success of Deep Learning. The authors conclude that DNNs effectively learn efficient representations that are approximate minimal sufficient statistics in the IB sense, and that the IB framework can provide new insights and algorithms for training DNNs.