This paper explores the benefits of overparameterization in deep learning models while addressing the computational challenges it introduces. The authors demonstrate that by leveraging the inherent low-dimensional structures of data and compressible dynamics within model parameters, it is possible to achieve the benefits of overparameterization without the associated computational burden. They apply this approach to deep low-rank matrix completion and language model fine-tuning, showing significant improvements in training efficiency and performance. The key insight is that the learning dynamics of each weight matrix in deep overparameterized models are confined to an invariant low-dimensional subspace. This allows for the construction and training of compact, highly compressed factorizations that retain the benefits of their overparameterized counterparts. In the context of deep matrix completion, this approach substantially improves training efficiency while maintaining the advantages of overparameterization. For language model fine-tuning, the authors propose a method called "Deep LoRA," which improves upon the existing low-rank adaptation (LoRA) technique by reducing overfitting and simplifying hyperparameter setup while maintaining comparable efficiency. Deep LoRA is shown to be effective on natural language tasks, particularly when fine-tuning with limited data. The method leverages the invariant low-dimensional subspaces identified in the learning dynamics, allowing for efficient parameter-efficient fine-tuning. The authors validate the effectiveness of Deep LoRA through extensive experiments, demonstrating its superiority over vanilla LoRA in terms of performance, robustness to hyperparameter choices, and efficiency. Theoretical analysis supports these findings, showing that the learning dynamics of deep overparameterized models are confined to low-dimensional subspaces, which can be exploited for efficient training. The results highlight the potential of compressible dynamics in deep learning for achieving efficient and effective model training and adaptation.This paper explores the benefits of overparameterization in deep learning models while addressing the computational challenges it introduces. The authors demonstrate that by leveraging the inherent low-dimensional structures of data and compressible dynamics within model parameters, it is possible to achieve the benefits of overparameterization without the associated computational burden. They apply this approach to deep low-rank matrix completion and language model fine-tuning, showing significant improvements in training efficiency and performance. The key insight is that the learning dynamics of each weight matrix in deep overparameterized models are confined to an invariant low-dimensional subspace. This allows for the construction and training of compact, highly compressed factorizations that retain the benefits of their overparameterized counterparts. In the context of deep matrix completion, this approach substantially improves training efficiency while maintaining the advantages of overparameterization. For language model fine-tuning, the authors propose a method called "Deep LoRA," which improves upon the existing low-rank adaptation (LoRA) technique by reducing overfitting and simplifying hyperparameter setup while maintaining comparable efficiency. Deep LoRA is shown to be effective on natural language tasks, particularly when fine-tuning with limited data. The method leverages the invariant low-dimensional subspaces identified in the learning dynamics, allowing for efficient parameter-efficient fine-tuning. The authors validate the effectiveness of Deep LoRA through extensive experiments, demonstrating its superiority over vanilla LoRA in terms of performance, robustness to hyperparameter choices, and efficiency. Theoretical analysis supports these findings, showing that the learning dynamics of deep overparameterized models are confined to low-dimensional subspaces, which can be exploited for efficient training. The results highlight the potential of compressible dynamics in deep learning for achieving efficient and effective model training and adaptation.