This paper introduces linear classifier probes as a method to understand the roles and dynamics of intermediate layers in deep neural networks. The probes are linear classifiers trained independently of the model to measure how suitable features at each layer are for classification. This approach helps in understanding the model's behavior, diagnosing potential issues, and gaining intuition about its training process. The authors apply this technique to popular models like Inception v3 and ResNet-50. They observe that the linear separability of features increases monotonically with depth, indicating that deeper layers contain more abstract and separable features. This finding suggests that deep neural networks have a "greedy" aspect in their representation learning, where deeper layers progressively refine the representation. The paper also discusses related work, including linear classification with kernel PCA, generalization and transferability of layers, relevance propagation, and SVCCA. These methods aim to interpret and understand neural networks, often focusing on feature visualization and relevance analysis. The authors highlight practical concerns such as the computational cost of using linear classifier probes and the need for dimensionality reduction when dealing with high-dimensional features. They also demonstrate how probes can be used to diagnose training issues, such as in the case of a pathologically deep model with a long skip connection. The paper concludes that linear classifier probes are a valuable tool for understanding deep neural networks, providing insights into the dynamics of intermediate layers and the effectiveness of features at different depths. The results suggest that the monotonic increase in linear separability with depth is a natural consequence of conventional training and not due to the probes themselves. This finding has implications for the design and optimization of deep neural networks.This paper introduces linear classifier probes as a method to understand the roles and dynamics of intermediate layers in deep neural networks. The probes are linear classifiers trained independently of the model to measure how suitable features at each layer are for classification. This approach helps in understanding the model's behavior, diagnosing potential issues, and gaining intuition about its training process. The authors apply this technique to popular models like Inception v3 and ResNet-50. They observe that the linear separability of features increases monotonically with depth, indicating that deeper layers contain more abstract and separable features. This finding suggests that deep neural networks have a "greedy" aspect in their representation learning, where deeper layers progressively refine the representation. The paper also discusses related work, including linear classification with kernel PCA, generalization and transferability of layers, relevance propagation, and SVCCA. These methods aim to interpret and understand neural networks, often focusing on feature visualization and relevance analysis. The authors highlight practical concerns such as the computational cost of using linear classifier probes and the need for dimensionality reduction when dealing with high-dimensional features. They also demonstrate how probes can be used to diagnose training issues, such as in the case of a pathologically deep model with a long skip connection. The paper concludes that linear classifier probes are a valuable tool for understanding deep neural networks, providing insights into the dynamics of intermediate layers and the effectiveness of features at different depths. The results suggest that the monotonic increase in linear separability with depth is a natural consequence of conventional training and not due to the probes themselves. This finding has implications for the design and optimization of deep neural networks.