Supervised Contrastive Learning

Supervised Contrastive Learning

10 Mar 2021 | Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, Dilip Krishnan
Supervised Contrastive Learning (SupCon) is a method that extends self-supervised contrastive learning to the fully supervised setting, leveraging label information to improve performance. The method uses a contrastive loss function that pulls together embeddings of samples from the same class while pushing apart embeddings of samples from different classes. This approach outperforms traditional contrastive losses such as triplet, max-margin, and N-pairs losses, and achieves state-of-the-art results on the ImageNet dataset with ResNet-200, achieving a top-1 accuracy of 81.4%, which is 0.8% higher than the best cross-entropy loss for this architecture. SupCon is also more robust to natural corruptions and is more stable to hyperparameter settings such as optimizers and data augmentations. The loss function is simple to implement and is available in TensorFlow code at https://t.ly/supcon. The method is effective for both self-supervised and supervised learning, and provides a unifying loss function that can be used for either. SupCon is shown to consistently outperform cross-entropy on various datasets and is more robust to natural corruptions. The method is also more stable to hyperparameter settings and is less sensitive to a range of hyperparameters. SupCon is a generalization of both the triplet and N-pair losses, and allows for multiple positives and negatives per anchor, leading to state-of-the-art performance without the need for hard negative mining. The method is effective for both self-supervised and supervised learning, and provides a unifying loss function that can be used for either. SupCon is shown to consistently outperform cross-entropy on various datasets and is more robust to natural corruptions. The method is also more stable to hyperparameter settings and is less sensitive to a range of hyperparameters. SupCon is a generalization of both the triplet and N-pair losses, and allows for multiple positives and negatives per anchor, leading to state-of-the-art performance without the need for hard negative mining.Supervised Contrastive Learning (SupCon) is a method that extends self-supervised contrastive learning to the fully supervised setting, leveraging label information to improve performance. The method uses a contrastive loss function that pulls together embeddings of samples from the same class while pushing apart embeddings of samples from different classes. This approach outperforms traditional contrastive losses such as triplet, max-margin, and N-pairs losses, and achieves state-of-the-art results on the ImageNet dataset with ResNet-200, achieving a top-1 accuracy of 81.4%, which is 0.8% higher than the best cross-entropy loss for this architecture. SupCon is also more robust to natural corruptions and is more stable to hyperparameter settings such as optimizers and data augmentations. The loss function is simple to implement and is available in TensorFlow code at https://t.ly/supcon. The method is effective for both self-supervised and supervised learning, and provides a unifying loss function that can be used for either. SupCon is shown to consistently outperform cross-entropy on various datasets and is more robust to natural corruptions. The method is also more stable to hyperparameter settings and is less sensitive to a range of hyperparameters. SupCon is a generalization of both the triplet and N-pair losses, and allows for multiple positives and negatives per anchor, leading to state-of-the-art performance without the need for hard negative mining. The method is effective for both self-supervised and supervised learning, and provides a unifying loss function that can be used for either. SupCon is shown to consistently outperform cross-entropy on various datasets and is more robust to natural corruptions. The method is also more stable to hyperparameter settings and is less sensitive to a range of hyperparameters. SupCon is a generalization of both the triplet and N-pair losses, and allows for multiple positives and negatives per anchor, leading to state-of-the-art performance without the need for hard negative mining.
Reach us at info@study.space
[slides] Supervised Contrastive Learning | StudySpace