This Perspective explores the impact of neural representations on the generalization capabilities of artificial and brain neural networks. It introduces two hypotheses: first, the geometric properties of neural manifolds associated with discrete cognitive entities are powerful order parameters that link the neural substrate to generalization capabilities. Second, the theory of learning in wide DNNs provides mechanistic insights into the learning processes generating desired neural representational geometries and generalization. The study discusses the role of weight norm regularization, network architecture, and hyperparameters in this process. It also examines the dynamics of learning and its relevance to representational drift in the brain. The paper reviews recent progress in studying the geometry of neural manifolds, particularly in visual object recognition, and discusses theories connecting manifold dimension and radius to generalization capacity. It also explores the role of geometric properties in few-shot learning and the alignment between visual and language representations. The paper further discusses the theory of deep learning, including the thermodynamic limit and finite width kernel renormalization, and the implications for feature learning in wide networks. The study highlights the importance of representational geometry in generalization and the role of learning dynamics in shaping neural representations. The paper concludes with a discussion of the implications of these findings for understanding both biological and artificial neural networks.This Perspective explores the impact of neural representations on the generalization capabilities of artificial and brain neural networks. It introduces two hypotheses: first, the geometric properties of neural manifolds associated with discrete cognitive entities are powerful order parameters that link the neural substrate to generalization capabilities. Second, the theory of learning in wide DNNs provides mechanistic insights into the learning processes generating desired neural representational geometries and generalization. The study discusses the role of weight norm regularization, network architecture, and hyperparameters in this process. It also examines the dynamics of learning and its relevance to representational drift in the brain. The paper reviews recent progress in studying the geometry of neural manifolds, particularly in visual object recognition, and discusses theories connecting manifold dimension and radius to generalization capacity. It also explores the role of geometric properties in few-shot learning and the alignment between visual and language representations. The paper further discusses the theory of deep learning, including the thermodynamic limit and finite width kernel renormalization, and the implications for feature learning in wide networks. The study highlights the importance of representational geometry in generalization and the role of learning dynamics in shaping neural representations. The paper concludes with a discussion of the implications of these findings for understanding both biological and artificial neural networks.