This paper investigates the role of memorization in deep learning, focusing on how deep neural networks (DNNs) process noise data versus real data. The authors explore the differences in gradient-based optimization between these two types of data and demonstrate that DNNs can memorize noise without needing extensive training. However, they also show that DNNs prioritize learning simple patterns before memorizing, which suggests that they are not simply memorizing data but instead learning to extract meaningful patterns. The study highlights qualitative differences in DNN optimization on real data versus noise data. For instance, DNNs trained on real data show a higher sensitivity to certain examples, indicating that they are learning to recognize patterns rather than memorizing individual instances. Additionally, the paper shows that regularization techniques can hinder memorization in DNNs without affecting their ability to learn from real data. The authors also examine the effective capacity of DNNs, which refers to the set of hypotheses that can be reached by applying a learning algorithm on a dataset. They argue that traditional learning theory may not fully explain the generalization performance of DNNs, as the training data itself plays a significant role in determining the degree of memorization. The paper presents experiments on MNIST and CIFAR10 datasets, comparing the performance of DNNs trained on real data versus noise data. The results show that DNNs learn patterns first, which allows them to generalize better on real data. The study also demonstrates that regularization techniques such as dropout can reduce memorization without compromising generalization. Overall, the paper suggests that DNNs do not simply memorize data but instead learn to extract meaningful patterns, which allows them to generalize well on real data. The findings have implications for understanding the behavior of deep learning models and the role of regularization in controlling memorization.This paper investigates the role of memorization in deep learning, focusing on how deep neural networks (DNNs) process noise data versus real data. The authors explore the differences in gradient-based optimization between these two types of data and demonstrate that DNNs can memorize noise without needing extensive training. However, they also show that DNNs prioritize learning simple patterns before memorizing, which suggests that they are not simply memorizing data but instead learning to extract meaningful patterns. The study highlights qualitative differences in DNN optimization on real data versus noise data. For instance, DNNs trained on real data show a higher sensitivity to certain examples, indicating that they are learning to recognize patterns rather than memorizing individual instances. Additionally, the paper shows that regularization techniques can hinder memorization in DNNs without affecting their ability to learn from real data. The authors also examine the effective capacity of DNNs, which refers to the set of hypotheses that can be reached by applying a learning algorithm on a dataset. They argue that traditional learning theory may not fully explain the generalization performance of DNNs, as the training data itself plays a significant role in determining the degree of memorization. The paper presents experiments on MNIST and CIFAR10 datasets, comparing the performance of DNNs trained on real data versus noise data. The results show that DNNs learn patterns first, which allows them to generalize better on real data. The study also demonstrates that regularization techniques such as dropout can reduce memorization without compromising generalization. Overall, the paper suggests that DNNs do not simply memorize data but instead learn to extract meaningful patterns, which allows them to generalize well on real data. The findings have implications for understanding the behavior of deep learning models and the role of regularization in controlling memorization.