This paper explores the structure of neural loss functions and the effect of loss landscapes on generalization using visualization methods. The authors introduce a "filter normalization" technique to visualize loss function curvature and compare different loss functions. They find that network architecture significantly affects the loss landscape, and training parameters influence the shape of minimizers. The study reveals that skip connections promote flat minimizers and prevent the transition to chaotic behavior, which helps explain why skip connections are necessary for training deep networks. The authors also observe that as networks become deeper, their loss landscapes transition from nearly convex to highly chaotic, which correlates with a drop in generalization error. They quantify non-convexity by calculating the smallest eigenvalues of the Hessian around local minima and visualize the results as heat maps. The study also shows that optimization trajectories lie in an extremely low-dimensional space, which can be explained by the presence of large, nearly convex regions in the loss landscape. The authors conclude that the geometry of neural loss functions plays a crucial role in generalization, and that filter normalization is a natural way to visualize loss function geometry. The study highlights the importance of network architecture, optimizer selection, and batch size in neural network training.This paper explores the structure of neural loss functions and the effect of loss landscapes on generalization using visualization methods. The authors introduce a "filter normalization" technique to visualize loss function curvature and compare different loss functions. They find that network architecture significantly affects the loss landscape, and training parameters influence the shape of minimizers. The study reveals that skip connections promote flat minimizers and prevent the transition to chaotic behavior, which helps explain why skip connections are necessary for training deep networks. The authors also observe that as networks become deeper, their loss landscapes transition from nearly convex to highly chaotic, which correlates with a drop in generalization error. They quantify non-convexity by calculating the smallest eigenvalues of the Hessian around local minima and visualize the results as heat maps. The study also shows that optimization trajectories lie in an extremely low-dimensional space, which can be explained by the presence of large, nearly convex regions in the loss landscape. The authors conclude that the geometry of neural loss functions plays a crucial role in generalization, and that filter normalization is a natural way to visualize loss function geometry. The study highlights the importance of network architecture, optimizer selection, and batch size in neural network training.