A Simple Framework for Contrastive Learning of Visual Representations

A Simple Framework for Contrastive Learning of Visual Representations

1 Jul 2020 | Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton
This paper introduces SimCLR, a simple framework for contrastive learning of visual representations. SimCLR simplifies previously proposed contrastive self-supervised learning algorithms by eliminating the need for specialized architectures or a memory bank. The authors systematically study the major components of the SimCLR framework to understand what enables effective contrastive prediction tasks. They find that: 1. Composition of multiple data augmentations is crucial for defining effective predictive tasks. 2. Introducing a learnable nonlinear transformation between the representation and the contrastive loss significantly improves the quality of learned representations. 3. Contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, SimCLR outperforms previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on SimCLR's self-supervised representations achieves 76.5% top-1 accuracy, a 7% relative improvement over previous state-of-the-art methods. When fine-tuned on only 1% of the labels, SimCLR achieves 85.8% top-5 accuracy, outperforming AlexNet with 100× fewer labels.This paper introduces SimCLR, a simple framework for contrastive learning of visual representations. SimCLR simplifies previously proposed contrastive self-supervised learning algorithms by eliminating the need for specialized architectures or a memory bank. The authors systematically study the major components of the SimCLR framework to understand what enables effective contrastive prediction tasks. They find that: 1. Composition of multiple data augmentations is crucial for defining effective predictive tasks. 2. Introducing a learnable nonlinear transformation between the representation and the contrastive loss significantly improves the quality of learned representations. 3. Contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, SimCLR outperforms previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on SimCLR's self-supervised representations achieves 76.5% top-1 accuracy, a 7% relative improvement over previous state-of-the-art methods. When fine-tuned on only 1% of the labels, SimCLR achieves 85.8% top-5 accuracy, outperforming AlexNet with 100× fewer labels.
Reach us at info@study.space
[slides and audio] A Simple Framework for Contrastive Learning of Visual Representations