This report provides a comprehensive overview of deep learning in neural networks, tracing the historical development and key contributions from the previous millennium. It distinguishes between shallow and deep learners based on the depth of their credit assignment paths (CAPs), which are chains of causal links between actions and effects. The report covers supervised learning, unsupervised learning, reinforcement learning, and evolutionary computation, highlighting the role of CAPs in these areas. It reviews the history of backpropagation, the development of deep networks, and the challenges and solutions encountered in training deep neural networks. The report also discusses the use of GPUs for accelerating deep learning and the importance of hierarchical representations and Occam's Razor in achieving efficient and effective deep learning. Finally, it explores the potential of deep learning in spiking neurons and its implications for neuroscience.This report provides a comprehensive overview of deep learning in neural networks, tracing the historical development and key contributions from the previous millennium. It distinguishes between shallow and deep learners based on the depth of their credit assignment paths (CAPs), which are chains of causal links between actions and effects. The report covers supervised learning, unsupervised learning, reinforcement learning, and evolutionary computation, highlighting the role of CAPs in these areas. It reviews the history of backpropagation, the development of deep networks, and the challenges and solutions encountered in training deep neural networks. The report also discusses the use of GPUs for accelerating deep learning and the importance of hierarchical representations and Occam's Razor in achieving efficient and effective deep learning. Finally, it explores the potential of deep learning in spiking neurons and its implications for neuroscience.