Learning to Reweight Examples for Robust Deep Learning

Learning to Reweight Examples for Robust Deep Learning

5 May 2019 | Mengye Ren, Wenyuan Zeng, Bin Yang, Raquel Urtasun
Deep neural networks (DNNs) are powerful tools for supervised learning tasks, but they can easily overfit to training set biases and label noises. Example reweighting algorithms are popular solutions to these problems, but they require careful tuning of hyperparameters. This work proposes a novel meta-learning algorithm that learns to assign weights to training examples based on their gradient directions. The method performs a meta gradient descent step on the current mini-batch example weights to minimize the loss on a clean, unbiased validation set. The proposed method is easy to implement on any type of deep network, does not require additional hyperparameter tuning, and achieves impressive performance on class imbalance and corrupted label problems, even with limited clean validation data. The paper includes a detailed derivation of the algorithm, experimental results on MNIST and CIFAR datasets, and theoretical guarantees of convergence.Deep neural networks (DNNs) are powerful tools for supervised learning tasks, but they can easily overfit to training set biases and label noises. Example reweighting algorithms are popular solutions to these problems, but they require careful tuning of hyperparameters. This work proposes a novel meta-learning algorithm that learns to assign weights to training examples based on their gradient directions. The method performs a meta gradient descent step on the current mini-batch example weights to minimize the loss on a clean, unbiased validation set. The proposed method is easy to implement on any type of deep network, does not require additional hyperparameter tuning, and achieves impressive performance on class imbalance and corrupted label problems, even with limited clean validation data. The paper includes a detailed derivation of the algorithm, experimental results on MNIST and CIFAR datasets, and theoretical guarantees of convergence.
Reach us at info@study.space