This paper reviews the application of non-volatile memory (NVM) devices in neuromorphic computing, focusing on three computing paradigms: spiking neural networks (SNNs), deep neural networks (DNNs), and 'Memcomputing'. SNNs use NVM synaptic connections updated by local learning rules like spike-timing-dependent-plasticity (STDP), inspired by biological processes. DNNs can represent matrices of synaptic weights using NVM arrays, enabling efficient matrix-vector multiplication and backpropagation. The paper surveys various NVM devices, including phase change memory (PCM), conductive-bridging RAM (CBRAM), filamentary and non-filamentary resistive RAM (RRAM), and others, discussing their properties and limitations. It also explores the use of NVM devices as synapses or neurons in neuromorphic applications, highlighting the challenges and potential solutions for implementing these devices in crossbar arrays. The paper concludes with a discussion on the energy efficiency and performance improvements that NVM-based systems can offer compared to traditional Von Neumann architecture.This paper reviews the application of non-volatile memory (NVM) devices in neuromorphic computing, focusing on three computing paradigms: spiking neural networks (SNNs), deep neural networks (DNNs), and 'Memcomputing'. SNNs use NVM synaptic connections updated by local learning rules like spike-timing-dependent-plasticity (STDP), inspired by biological processes. DNNs can represent matrices of synaptic weights using NVM arrays, enabling efficient matrix-vector multiplication and backpropagation. The paper surveys various NVM devices, including phase change memory (PCM), conductive-bridging RAM (CBRAM), filamentary and non-filamentary resistive RAM (RRAM), and others, discussing their properties and limitations. It also explores the use of NVM devices as synapses or neurons in neuromorphic applications, highlighting the challenges and potential solutions for implementing these devices in crossbar arrays. The paper concludes with a discussion on the energy efficiency and performance improvements that NVM-based systems can offer compared to traditional Von Neumann architecture.