Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks **Authors:** Tim Salimans, Diederik P. Kingma **Institution:** OpenAI **Abstract:** This paper introduces *weight normalization*, a reparameterization technique for neural network weights that decouples the length of weight vectors from their direction. By reparameterizing the weights in this way, the optimization problem becomes better conditioned, leading to faster convergence of stochastic gradient descent. Inspired by batch normalization, weight normalization does not introduce dependencies between examples in a minibatch, making it suitable for recurrent models and noise-sensitive applications like deep reinforcement learning and generative models. Despite its simplicity, weight normalization achieves similar speed-ups to full batch normalization while having lower computational overhead. The method is demonstrated to be effective in various applications, including supervised image recognition, generative modeling, and deep reinforcement learning. **Introduction:** Recent successes in deep learning have highlighted the importance of optimizing the curvature of the objective function. Weight normalization addresses this by reparameterizing the weight vectors to fix their Euclidean norm, improving the conditioning of the optimization problem. This method is inspired by batch normalization but is deterministic and does not add noise to gradients. It is also computationally efficient, requiring no additional memory or significant extra computation. **Weight Normalization:** The paper proposes a new parameterization where the weight vector \(\mathbf{w}\) is expressed as \(\mathbf{w} = \frac{g}{\|\mathbf{v}\|} \mathbf{v}\), where \(\mathbf{v}\) is a unit vector and \(g\) is a scalar. This reparameterization ensures that the norm of \(\mathbf{w}\) is fixed, improving optimization performance. The method is shown to be robust to learning rates and can be applied to various neural network architectures, including RNNs and LSTMs. **Experiments:** The authors validate weight normalization on four different models: supervised classification (CIFAR-10), generative modeling (convolutional VAEs and DRAW), and reinforcement learning (DQN). Results show that weight normalization consistently improves convergence and performance compared to standard parameterization and batch normalization. The method is particularly effective in noise-sensitive applications and deep reinforcement learning, where batch normalization is less suitable. **Conclusion:** Weight normalization is a simple and effective reparameterization technique that accelerates the training of deep neural networks. It is easy to implement, has low computational overhead, and does not introduce dependencies between minibatch examples, making it a valuable tool for developing new deep learning architectures.Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks **Authors:** Tim Salimans, Diederik P. Kingma **Institution:** OpenAI **Abstract:** This paper introduces *weight normalization*, a reparameterization technique for neural network weights that decouples the length of weight vectors from their direction. By reparameterizing the weights in this way, the optimization problem becomes better conditioned, leading to faster convergence of stochastic gradient descent. Inspired by batch normalization, weight normalization does not introduce dependencies between examples in a minibatch, making it suitable for recurrent models and noise-sensitive applications like deep reinforcement learning and generative models. Despite its simplicity, weight normalization achieves similar speed-ups to full batch normalization while having lower computational overhead. The method is demonstrated to be effective in various applications, including supervised image recognition, generative modeling, and deep reinforcement learning. **Introduction:** Recent successes in deep learning have highlighted the importance of optimizing the curvature of the objective function. Weight normalization addresses this by reparameterizing the weight vectors to fix their Euclidean norm, improving the conditioning of the optimization problem. This method is inspired by batch normalization but is deterministic and does not add noise to gradients. It is also computationally efficient, requiring no additional memory or significant extra computation. **Weight Normalization:** The paper proposes a new parameterization where the weight vector \(\mathbf{w}\) is expressed as \(\mathbf{w} = \frac{g}{\|\mathbf{v}\|} \mathbf{v}\), where \(\mathbf{v}\) is a unit vector and \(g\) is a scalar. This reparameterization ensures that the norm of \(\mathbf{w}\) is fixed, improving optimization performance. The method is shown to be robust to learning rates and can be applied to various neural network architectures, including RNNs and LSTMs. **Experiments:** The authors validate weight normalization on four different models: supervised classification (CIFAR-10), generative modeling (convolutional VAEs and DRAW), and reinforcement learning (DQN). Results show that weight normalization consistently improves convergence and performance compared to standard parameterization and batch normalization. The method is particularly effective in noise-sensitive applications and deep reinforcement learning, where batch normalization is less suitable. **Conclusion:** Weight normalization is a simple and effective reparameterization technique that accelerates the training of deep neural networks. It is easy to implement, has low computational overhead, and does not introduce dependencies between minibatch examples, making it a valuable tool for developing new deep learning architectures.