The paper introduces a new architecture called highway networks, designed to facilitate the training of very deep neural networks. Highway networks use gating units to regulate the flow of information through multiple layers, allowing for unimpeded information flow across these layers. This mechanism, inspired by Long Short Term Memory (LSTM) recurrent neural networks, enables the optimization of networks with hundreds of layers using simple Stochastic Gradient Descent (SGD) and various activation functions. Preliminary experiments show that highway networks can be optimized with up to 900 layers, while traditional networks struggle with increasing depth. The authors also compare highway networks to Fitnets, finding that highway networks achieve similar or better accuracy on the CIFAR-10 dataset without the need for a pre-trained teacher network. The paper concludes by highlighting the potential of highway networks for studying the impact of depth on complex problems and the use of different activation functions in deep networks.The paper introduces a new architecture called highway networks, designed to facilitate the training of very deep neural networks. Highway networks use gating units to regulate the flow of information through multiple layers, allowing for unimpeded information flow across these layers. This mechanism, inspired by Long Short Term Memory (LSTM) recurrent neural networks, enables the optimization of networks with hundreds of layers using simple Stochastic Gradient Descent (SGD) and various activation functions. Preliminary experiments show that highway networks can be optimized with up to 900 layers, while traditional networks struggle with increasing depth. The authors also compare highway networks to Fitnets, finding that highway networks achieve similar or better accuracy on the CIFAR-10 dataset without the need for a pre-trained teacher network. The paper concludes by highlighting the potential of highway networks for studying the impact of depth on complex problems and the use of different activation functions in deep networks.