3 Mar 2018 | Jaehoon Lee*, Yasaman Bahri*, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein
This paper explores the equivalence between deep neural networks and Gaussian processes (GPs) in the limit of infinite network width. The authors derive the exact correspondence between infinitely wide deep networks and GPs, providing a computationally efficient method to compute the covariance function for these GPs. They demonstrate that trained neural network accuracy approaches that of the corresponding GP with increasing layer width, and that GP uncertainty is strongly correlated with prediction error. The experiments on MNIST and CIFAR-10 datasets show that the best NNGP performance is competitive with that of neural networks trained with standard gradient-based approaches, often surpassing conventional training. The paper also discusses the relationship between the performance of NNGPs and recent theories of signal propagation in random neural networks.This paper explores the equivalence between deep neural networks and Gaussian processes (GPs) in the limit of infinite network width. The authors derive the exact correspondence between infinitely wide deep networks and GPs, providing a computationally efficient method to compute the covariance function for these GPs. They demonstrate that trained neural network accuracy approaches that of the corresponding GP with increasing layer width, and that GP uncertainty is strongly correlated with prediction error. The experiments on MNIST and CIFAR-10 datasets show that the best NNGP performance is competitive with that of neural networks trained with standard gradient-based approaches, often surpassing conventional training. The paper also discusses the relationship between the performance of NNGPs and recent theories of signal propagation in random neural networks.