Self-training with Noisy Student improves ImageNet classification

Self-training with Noisy Student improves ImageNet classification

19 Jun 2020 | Qizhe Xie* 1, Minh-Thang Luong1, Eduard Hovy2, Quoc V. Le1
The paper introduces Noisy Student Training, a semi-supervised learning approach that significantly improves the accuracy and robustness of ImageNet models. By leveraging a large amount of unlabeled data, the method achieves 88.4% top-1 accuracy on ImageNet, outperforming the previous state-of-the-art model by 2.0%. The approach involves training an EfficientNet model on labeled images as a teacher to generate pseudo-labels for 300M unlabeled images, and then training a larger EfficientNet as a student model on the combination of labeled and pseudo-labeled images. The process is iterated by treating the student as a teacher, improving the model's performance over multiple iterations. The key contributions include adding noise to the student model (e.g., dropout, stochastic depth, and data augmentation) and using a larger student model than the teacher. The method also enhances robustness on challenging datasets like ImageNet-A, ImageNet-C, and ImageNet-P, reducing mean corruption error and mean flip rate. The paper includes ablation studies to validate the effectiveness of noise and iterative training, demonstrating that Noisy Student Training is a powerful technique for improving the performance of deep learning models.The paper introduces Noisy Student Training, a semi-supervised learning approach that significantly improves the accuracy and robustness of ImageNet models. By leveraging a large amount of unlabeled data, the method achieves 88.4% top-1 accuracy on ImageNet, outperforming the previous state-of-the-art model by 2.0%. The approach involves training an EfficientNet model on labeled images as a teacher to generate pseudo-labels for 300M unlabeled images, and then training a larger EfficientNet as a student model on the combination of labeled and pseudo-labeled images. The process is iterated by treating the student as a teacher, improving the model's performance over multiple iterations. The key contributions include adding noise to the student model (e.g., dropout, stochastic depth, and data augmentation) and using a larger student model than the teacher. The method also enhances robustness on challenging datasets like ImageNet-A, ImageNet-C, and ImageNet-P, reducing mean corruption error and mean flip rate. The paper includes ablation studies to validate the effectiveness of noise and iterative training, demonstrating that Noisy Student Training is a powerful technique for improving the performance of deep learning models.
Reach us at info@study.space